2024-04-13T13:01:34.062 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-13T13:01:34.063 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} --owner scheduled_teuthology@teuthology --name teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008 -- /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/orig.config.yaml 2024-04-13T13:01:34.111 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-13T13:01:34.206 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008 branch: reef description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: ceph-qa@ceph.com first_in_suite: false job_id: '7650008' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 4f70a9a90881107996c8b7a3e89ba8b32624713c owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9619 sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 sleep_before_teardown: 0 subset: 90/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4f70a9a90881107996c8b7a3e89ba8b32624713c targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG//M3DRroy6RXlxQxY3rAUPaE9ctyve7QrEx+XePnci2VHICPl1SRaHiQ5hVKBv5+qaWfUZ/wjpNTtD3aY7/4U= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRJMbsaAkfeSQKoa4lOc+m7Cn7k5Jn7V7UwZkRynUaEfDiUib85HyYAqrj1IrJgBro/SyCaVsnB0Pgix2dZxZ8= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-09_22:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-13T13:01:34.206 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa; will attempt to use it 2024-04-13T13:01:34.207 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks 2024-04-13T13:01:34.207 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-13T13:01:34.208 INFO:teuthology.task.internal:Checking packages... 2024-04-13T13:01:34.231 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd540ebaca6b131a1dd560e7f69e024b133bbaa42' 2024-04-13T13:01:34.231 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-13T13:01:34.231 INFO:teuthology.packaging:ref: None 2024-04-13T13:01:34.231 INFO:teuthology.packaging:tag: None 2024-04-13T13:01:34.231 INFO:teuthology.packaging:branch: reef 2024-04-13T13:01:34.232 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-13T13:01:34.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-13T13:01:34.463 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1217.g7d36322c 2024-04-13T13:01:34.465 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-13T13:01:34.471 INFO:teuthology.task.internal:no buildpackages task found 2024-04-13T13:01:34.471 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-13T13:01:34.478 INFO:teuthology.task.internal:Saving configuration 2024-04-13T13:01:34.496 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-13T13:01:34.502 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T13:01:34.528 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 12:54:34.160147', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG//M3DRroy6RXlxQxY3rAUPaE9ctyve7QrEx+XePnci2VHICPl1SRaHiQ5hVKBv5+qaWfUZ/wjpNTtD3aY7/4U='} 2024-04-13T13:01:34.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 12:54:34.162148', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRJMbsaAkfeSQKoa4lOc+m7Cn7k5Jn7V7UwZkRynUaEfDiUib85HyYAqrj1IrJgBro/SyCaVsnB0Pgix2dZxZ8='} 2024-04-13T13:01:34.550 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-13T13:01:34.556 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-04-13T13:01:34.556 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-04-13T13:01:34.556 INFO:teuthology.run_tasks:Running task console_log... 2024-04-13T13:01:34.651 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa09a577040>, signals=[15]) 2024-04-13T13:01:34.652 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-13T13:01:34.658 INFO:teuthology.task.internal:Opening connections... 2024-04-13T13:01:34.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T13:01:34.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:01:34.754 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T13:01:34.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:01:34.844 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-13T13:01:34.851 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-04-13T13:01:34.868 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-04-13T13:01:34.868 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-13T13:01:34.924 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-04-13T13:01:34.925 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-13T13:01:34.925 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-13T13:01:34.925 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-13T13:01:34.925 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-04-13T13:01:34.942 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-04-13T13:01:34.958 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-04-13T13:01:34.959 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-04-13T13:01:35.015 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-04-13T13:01:35.016 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-13T13:01:35.016 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-04-13T13:01:35.016 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-13T13:01:35.016 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-13T13:01:35.016 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-13T13:01:35.016 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-04-13T13:01:35.033 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-13T13:01:35.044 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-13T13:01:35.078 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-13T13:01:35.078 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-04-13T13:01:35.081 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-04-13T13:01:35.096 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-13T13:01:35.130 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-13T13:01:35.131 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-04-13T13:01:35.137 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-04-13T13:01:35.275 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-13T13:01:35.332 INFO:teuthology.run_tasks:Running task kernel... 2024-04-13T13:01:35.348 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-13T13:01:35.348 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-13T13:01:35.348 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-13T13:01:35.349 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-13T13:01:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T13:01:35.365 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-04-13T13:01:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T13:01:35.366 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-13T13:01:35.422 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-361.el9.x86_64 2024-04-13T13:01:35.422 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-361.el9.x86_64 2024-04-13T13:01:35.422 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-04-13T13:01:35.424 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-361.el9.x86_64 2024-04-13T13:01:35.424 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-361.el9.x86_64 2024-04-13T13:01:35.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-04-13T13:01:37.612 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 15 MB/s | 8.1 MB 00:00 2024-04-13T13:01:37.664 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 16 MB/s | 8.1 MB 00:00 2024-04-13T13:01:39.263 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.5 MB 00:00 2024-04-13T13:01:39.339 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.5 MB 00:00 2024-04-13T13:01:40.455 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-13T13:01:40.653 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 8.5 MB/s | 8.0 MB 00:00 2024-04-13T13:01:43.223 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-13T13:01:49.740 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.2 MB 00:02 2024-04-13T13:01:51.182 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-13T13:01:51.304 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-04-13T13:01:52.994 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-13T13:01:55.371 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-13T13:01:56.830 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-13T13:01:58.159 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-13T13:01:58.432 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-13T13:02:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:02:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:02:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:02:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:02:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:02:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:02:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:02:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:02:00.816 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-13T13:02:00.816 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-13T13:02:00.927 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:02:00.937 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:02:00.937 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout:Install 4 Packages 2024-04-13T13:02:00.938 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:02:00.939 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 77 M 2024-04-13T13:02:00.939 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 123 M 2024-04-13T13:02:00.939 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 2.5 MB/s | 214 kB 00:00 2024-04-13T13:02:01.402 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 38 MB/s | 15 MB 00:00 2024-04-13T13:02:02.953 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-modules-core-5.14.0-435.el9.x86_6 15 MB/s | 28 MB 00:01 2024-04-13T13:02:03.444 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 14 MB/s | 34 MB 00:02 2024-04-13T13:02:03.445 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:02:03.445 INFO:teuthology.orchestra.run.smithi083.stdout:Total 31 MB/s | 77 MB 00:02 2024-04-13T13:02:03.574 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-04-13T13:02:04.119 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:02:04.475 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:02:04.475 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:02:05.499 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:02:05.500 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:02:05.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:02:05.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.833 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.834 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.835 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.836 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.837 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:02:05.837 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:02:06.181 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-13T13:02:06.181 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-13T13:02:06.296 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:02:06.301 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:02:06.301 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-04-13T13:02:06.302 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:02:06.303 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 77 M 2024-04-13T13:02:06.303 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 123 M 2024-04-13T13:02:06.303 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:02:07.642 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 1.5 MB/s | 214 kB 00:00 2024-04-13T13:02:07.943 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 35 MB/s | 15 MB 00:00 2024-04-13T13:02:07.971 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:02:08.376 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 39 MB/s | 34 MB 00:00 2024-04-13T13:02:08.574 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-13T13:02:08.662 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-13T13:02:08.709 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-core-5.14.0-435.el9.x86_6 26 MB/s | 28 MB 00:01 2024-04-13T13:02:08.710 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:02:08.710 INFO:teuthology.orchestra.run.smithi122.stdout:Total 32 MB/s | 77 MB 00:02 2024-04-13T13:02:09.377 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:02:09.756 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:02:09.756 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:02:10.747 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:02:10.747 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:02:11.418 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-13T13:02:11.617 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-13T13:02:13.211 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:02:14.058 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-13T13:02:14.137 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-13T13:02:16.350 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-13T13:02:16.393 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:16.906 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-13T13:02:17.053 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-13T13:02:21.088 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-13T13:02:21.887 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:26.615 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:50.551 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:50.552 INFO:teuthology.orchestra.run.smithi083.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-13T13:02:50.552 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:02:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:51.016 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:51.016 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-13T13:02:51.016 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-13T13:02:51.016 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-13T13:02:51.450 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:02:51.451 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:02:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-13T13:02:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-13T13:02:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-13T13:02:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-13T13:02:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:02:51.451 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:02:51.824 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-04-13T13:02:51.922 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-13T13:02:51.922 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-13T13:02:51.922 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-13T13:02:51.922 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-13T13:02:51.922 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-13T13:02:51.923 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-04-13T13:02:51.923 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-04-13T13:02:51.923 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-04-13T13:02:52.003 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-13T13:02:52.003 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-13T13:02:52.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-04-13T13:02:52.037 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-04-13T13:02:52.037 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-04-13T13:02:52.037 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.06 2024-04-13T13:02:52.037 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 68.el9 2024-04-13T13:02:52.037 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-04-13T13:02:52.037 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Group : Unspecified 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 8264254 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : builder@centos.org 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-04-13T13:02:52.038 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:02:52.039 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-13T13:02:52.039 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-13T13:02:52.039 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-13T13:02:52.039 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-04-13T13:02:52.039 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:02:52.039 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-04-13T13:02:52.040 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-13T13:02:52.040 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-435.el9.x86_64 2024-04-13T13:02:52.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-13T13:02:52.717 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-04-13T13:02:54.164 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-13T13:02:54.198 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-04-13T13:02:54.201 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-04-13T13:02:54.257 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.nr70ljiOs0 2024-04-13T13:02:54.257 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nr70ljiOs0 2024-04-13T13:02:54.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.nr70ljiOs0 2024-04-13T13:02:54.473 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.nr70ljiOs0 is 6KB 2024-04-13T13:02:54.523 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.nr70ljiOs0 2024-04-13T13:02:54.537 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-04-13T13:02:54.601 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-13T13:02:54.602 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-13T13:02:54.602 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-13T13:02:54.602 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-13T13:02:54.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-13T13:02:54.767 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-04-13T13:02:56.012 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:56.013 INFO:teuthology.orchestra.run.smithi122.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-13T13:02:56.013 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:02:56.059 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:56.448 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:56.448 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-13T13:02:56.449 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-13T13:02:56.449 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:02:56.882 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:02:57.217 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-04-13T13:02:57.318 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-13T13:02:57.319 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-13T13:02:57.319 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-13T13:02:57.319 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-13T13:02:57.319 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-13T13:02:57.319 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-04-13T13:02:57.319 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-04-13T13:02:57.319 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-04-13T13:02:57.398 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-13T13:02:57.398 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-13T13:02:57.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-04-13T13:02:57.442 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 68.el9 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8264254 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-13T13:02:57.443 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-04-13T13:02:57.444 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-04-13T13:02:57.444 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:02:57.444 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-13T13:02:57.444 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-13T13:02:57.444 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-13T13:02:57.444 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-04-13T13:02:57.444 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:02:57.444 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-04-13T13:02:57.445 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-13T13:02:57.445 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-435.el9.x86_64 2024-04-13T13:02:57.445 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-13T13:02:58.011 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-04-13T13:02:59.439 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-13T13:02:59.466 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-04-13T13:02:59.468 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-04-13T13:02:59.510 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.WBAlUD5bUi 2024-04-13T13:02:59.510 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WBAlUD5bUi 2024-04-13T13:02:59.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.WBAlUD5bUi 2024-04-13T13:02:59.731 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.WBAlUD5bUi is 6KB 2024-04-13T13:02:59.781 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.WBAlUD5bUi 2024-04-13T13:02:59.795 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-04-13T13:02:59.860 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-13T13:02:59.860 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-13T13:02:59.860 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-13T13:02:59.860 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-13T13:02:59.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-13T13:02:59.998 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-04-13T13:03:24.794 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-04-13T13:03:24.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-04-13T13:03:24.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:03:30.006 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-04-13T13:03:30.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-13T13:03:30.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:03:43.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-04-13T13:03:52.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-04-13T13:03:52.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:03:55.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-04-13T13:04:07.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-04-13T13:04:07.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:04:30.031 DEBUG:teuthology.orchestra.remote:timed out 2024-04-13T13:04:39.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-13T13:04:39.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:04:39.128 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-04-13T13:04:39.425 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-13T13:04:39.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-04-13T13:04:39.518 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-435.el9.x86_64"... 2024-04-13T13:04:39.518 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-04-13T13:04:39.532 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-435.el9.x86_64 2024-04-13T13:04:39.533 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-13T13:04:39.533 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-13T13:04:39.533 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-13T13:04:39.904 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-13T13:04:39.904 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-435.el9.x86_64"... 2024-04-13T13:04:39.904 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-13T13:04:39.918 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-435.el9.x86_64 2024-04-13T13:04:39.918 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-13T13:04:39.918 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-13T13:04:39.918 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-13T13:04:40.534 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-04-13T13:04:40.534 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-13T13:04:40.612 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-04-13T13:04:40.638 DEBUG:teuthology.parallel:result is None 2024-04-13T13:04:40.919 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-04-13T13:04:40.919 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-13T13:04:41.030 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-04-13T13:04:41.083 DEBUG:teuthology.parallel:result is None 2024-04-13T13:04:41.084 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-13T13:04:41.094 INFO:teuthology.task.internal:Creating test directory... 2024-04-13T13:04:41.094 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-13T13:04:41.097 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-13T13:04:41.111 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-13T13:04:41.121 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-13T13:04:41.169 INFO:teuthology.task.internal:Creating archive directory... 2024-04-13T13:04:41.170 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-13T13:04:41.172 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-13T13:04:41.201 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-13T13:04:41.208 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-13T13:04:41.208 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-13T13:04:41.234 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-13T13:04:41.259 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T13:04:41.272 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T13:04:41.284 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T13:04:41.296 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T13:04:41.298 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-13T13:04:41.306 INFO:teuthology.task.internal:Configuring sudo... 2024-04-13T13:04:41.306 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-13T13:04:41.319 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-13T13:04:41.345 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-13T13:04:41.355 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-13T13:04:41.355 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-13T13:04:41.389 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-13T13:04:41.403 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T13:04:41.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T13:04:41.537 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T13:04:41.623 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T13:04:41.671 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:04:41.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-13T13:04:41.742 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T13:04:41.782 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T13:04:41.835 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T13:04:41.912 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T13:04:41.960 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:04:41.960 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-13T13:04:42.028 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-04-13T13:04:42.045 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-04-13T13:04:42.089 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T13:04:42.146 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T13:04:42.530 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-13T13:04:42.539 INFO:teuthology.task.internal:Starting timer... 2024-04-13T13:04:42.539 INFO:teuthology.run_tasks:Running task pcp... 2024-04-13T13:04:42.562 INFO:teuthology.run_tasks:Running task selinux... 2024-04-13T13:04:42.593 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-04-13T13:04:42.697 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-04-13T13:04:42.700 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-04-13T13:04:42.766 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-04-13T13:04:42.768 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-13T13:04:42.768 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-04-13T13:04:42.806 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-04-13T13:04:42.807 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-04-13T13:04:42.835 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-04-13T13:04:42.836 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-04-13T13:04:42.836 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T13:04:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T13:04:42.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T13:04:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T13:04:42.922 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-13T13:04:42.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-04-13T13:04:43.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-04-13T13:04:43.154 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-13T13:04:43.163 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-13T13:04:43.170 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-13T13:04:43.295 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-13T13:04:43.311 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-13T13:04:43.312 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-13T13:08:59.411 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-04-13T13:08:59.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-04-13T13:08:59.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:08:59.506 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-04-13T13:08:59.580 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-04-13T13:08:59.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-13T13:08:59.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T13:08:59.670 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-13T13:08:59.753 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-13T13:08:59.753 INFO:teuthology.run_tasks:Running task clock... 2024-04-13T13:08:59.765 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-13T13:08:59.765 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-13T13:08:59.766 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T13:08:59.768 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-13T13:08:59.768 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T13:08:59.797 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-13T13:08:59.814 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-13T13:08:59.824 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-13T13:08:59.840 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-13T13:08:59.843 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-04-13T13:08:59.856 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-04-13T13:08:59.869 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-04-13T13:08:59.872 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-13T13:08:59.882 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-04-13T13:08:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-13T13:08:59.899 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-13T13:08:59.915 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-13T13:08:59.937 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-04-13T13:08:59.940 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T13:08:59.940 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-04-13T13:08:59.941 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T13:08:59.941 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T13:08:59.941 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T13:08:59.941 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T13:08:59.965 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-13T13:08:59.968 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T13:08:59.968 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-13T13:08:59.968 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T13:08:59.968 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T13:08:59.968 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T13:08:59.968 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T13:08:59.969 INFO:teuthology.run_tasks:Running task install... 2024-04-13T13:08:59.977 DEBUG:teuthology.task.install:project ceph 2024-04-13T13:08:59.978 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}} 2024-04-13T13:08:59.978 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'} 2024-04-13T13:08:59.978 INFO:teuthology.task.install:Using flavor: default 2024-04-13T13:08:59.984 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-13T13:08:59.984 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-04-13T13:08:59.984 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-13T13:08:59.985 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-13T13:08:59.987 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-13T13:08:59.987 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-13T13:09:00.134 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/9/flavors/default/ 2024-04-13T13:09:00.135 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1187.gd540ebac 2024-04-13T13:09:00.217 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/9/flavors/default/ 2024-04-13T13:09:00.217 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1187.gd540ebac 2024-04-13T13:09:00.251 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-13T13:09:00.251 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:09:00.251 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-13T13:09:00.279 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-04-13T13:09:00.280 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d540ebaca6b131a1dd560e7f69e024b133bbaa42/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-13T13:09:00.325 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-13T13:09:00.325 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:09:00.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-13T13:09:00.348 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-13T13:09:00.353 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-04-13T13:09:00.353 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d540ebaca6b131a1dd560e7f69e024b133bbaa42/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-13T13:09:00.421 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-13T13:09:00.427 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-13T13:09:00.493 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-04-13T13:09:00.495 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-04-13T13:09:00.501 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-13T13:09:00.567 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-04-13T13:09:00.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-04-13T13:09:00.751 INFO:teuthology.orchestra.run.smithi122.stdout:74 files removed 2024-04-13T13:09:00.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-04-13T13:09:00.808 INFO:teuthology.orchestra.run.smithi083.stdout:74 files removed 2024-04-13T13:09:00.834 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2024-04-13T13:09:01.571 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 155 kB/s | 77 kB 00:00 2024-04-13T13:09:01.573 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 175 kB/s | 77 kB 00:00 2024-04-13T13:09:02.005 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-04-13T13:09:02.007 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-04-13T13:09:02.481 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-04-13T13:09:02.516 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-04-13T13:09:03.142 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 13 MB/s | 8.1 MB 00:00 2024-04-13T13:09:03.152 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-13T13:09:05.033 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.5 MB 00:00 2024-04-13T13:09:05.128 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.5 MB 00:00 2024-04-13T13:09:08.643 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-04-13T13:09:10.400 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-04-13T13:09:14.501 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-13T13:09:14.859 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 887 kB/s | 8.0 MB 00:09 2024-04-13T13:09:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-04-13T13:09:17.920 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-13T13:09:17.959 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 19 MB 00:01 2024-04-13T13:09:22.485 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.2 MB 00:00 2024-04-13T13:09:23.078 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-13T13:09:24.328 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-13T13:09:25.653 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.654 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:09:25.655 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.656 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.657 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.658 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.659 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.660 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.661 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.662 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.663 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.663 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:09:25.663 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.663 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.663 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:09:25.663 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:09:26.163 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 7.7 M 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 3.3 M 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 3.0 M 2024-04-13T13:09:26.172 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 5.1 M 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 18 M 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 25 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 709 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 474 k 2024-04-13T13:09:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 4.5 M 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 45 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 129 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 161 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 320 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 299 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 99 k 2024-04-13T13:09:26.174 INFO:teuthology.orchestra.run.smithi122.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout:Install 28 Packages 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 53 M 2024-04-13T13:09:26.175 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:09:26.221 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-13T13:09:26.954 INFO:teuthology.orchestra.run.smithi122.stdout:(1/30): ceph-base-18.2.2-1187.gd540ebac.el9.x86 9.3 MB/s | 5.1 MB 00:00 2024-04-13T13:09:26.996 INFO:teuthology.orchestra.run.smithi122.stdout:(2/30): ceph-selinux-18.2.2-1187.gd540ebac.el9. 607 kB/s | 25 kB 00:00 2024-04-13T13:09:27.080 INFO:teuthology.orchestra.run.smithi122.stdout:(3/30): libcephfs2-18.2.2-1187.gd540ebac.el9.x8 8.3 MB/s | 709 kB 00:00 2024-04-13T13:09:27.147 INFO:teuthology.orchestra.run.smithi122.stdout:(4/30): libradosstriper1-18.2.2-1187.gd540ebac. 7.0 MB/s | 474 kB 00:00 2024-04-13T13:09:27.500 INFO:teuthology.orchestra.run.smithi122.stdout:(5/30): librgw2-18.2.2-1187.gd540ebac.el9.x86_6 13 MB/s | 4.5 MB 00:00 2024-04-13T13:09:27.555 INFO:teuthology.orchestra.run.smithi122.stdout:(6/30): python3-ceph-argparse-18.2.2-1187.gd540 819 kB/s | 45 kB 00:00 2024-04-13T13:09:27.622 INFO:teuthology.orchestra.run.smithi122.stdout:(7/30): python3-ceph-common-18.2.2-1187.gd540eb 1.9 MB/s | 129 kB 00:00 2024-04-13T13:09:27.690 INFO:teuthology.orchestra.run.smithi122.stdout:(8/30): python3-cephfs-18.2.2-1187.gd540ebac.el 2.4 MB/s | 161 kB 00:00 2024-04-13T13:09:27.764 INFO:teuthology.orchestra.run.smithi122.stdout:(9/30): python3-rados-18.2.2-1187.gd540ebac.el9 4.2 MB/s | 320 kB 00:00 2024-04-13T13:09:27.849 INFO:teuthology.orchestra.run.smithi122.stdout:(10/30): python3-rbd-18.2.2-1187.gd540ebac.el9. 3.5 MB/s | 299 kB 00:00 2024-04-13T13:09:27.907 INFO:teuthology.orchestra.run.smithi122.stdout:(11/30): python3-rgw-18.2.2-1187.gd540ebac.el9. 1.7 MB/s | 99 kB 00:00 2024-04-13T13:09:27.999 INFO:teuthology.orchestra.run.smithi122.stdout:(12/30): ceph-radosgw-18.2.2-1187.gd540ebac.el9 4.8 MB/s | 7.7 MB 00:01 2024-04-13T13:09:28.124 INFO:teuthology.orchestra.run.smithi122.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 492 kB/s | 107 kB 00:00 2024-04-13T13:09:28.149 INFO:teuthology.orchestra.run.smithi122.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-04-13T13:09:28.166 INFO:teuthology.orchestra.run.smithi122.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-13T13:09:28.233 INFO:teuthology.orchestra.run.smithi122.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 662 kB 00:00 2024-04-13T13:09:28.259 INFO:teuthology.orchestra.run.smithi122.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-04-13T13:09:28.276 INFO:teuthology.orchestra.run.smithi122.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-04-13T13:09:28.584 INFO:teuthology.orchestra.run.smithi122.stdout:(19/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-04-13T13:09:28.609 INFO:teuthology.orchestra.run.smithi122.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 316 kB/s | 111 kB 00:00 2024-04-13T13:09:28.668 INFO:teuthology.orchestra.run.smithi122.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-04-13T13:09:28.751 INFO:teuthology.orchestra.run.smithi122.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 584 kB/s | 49 kB 00:00 2024-04-13T13:09:28.877 INFO:teuthology.orchestra.run.smithi122.stdout:(23/30): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-13T13:09:29.052 INFO:teuthology.orchestra.run.smithi122.stdout:(24/30): ceph-common-18.2.2-1187.gd540ebac.el9. 6.9 MB/s | 18 MB 00:02 2024-04-13T13:09:29.077 INFO:teuthology.orchestra.run.smithi122.stdout:(25/30): libunwind-1.6.2-1.el9.x86_64.rpm 207 kB/s | 67 kB 00:00 2024-04-13T13:09:29.111 INFO:teuthology.orchestra.run.smithi122.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-04-13T13:09:29.170 INFO:teuthology.orchestra.run.smithi122.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-13T13:09:29.445 INFO:teuthology.orchestra.run.smithi122.stdout:(28/30): librbd1-18.2.2-1187.gd540ebac.el9.x86_ 11 MB/s | 3.0 MB 00:00 2024-04-13T13:09:29.512 INFO:teuthology.orchestra.run.smithi122.stdout:(29/30): librados2-18.2.2-1187.gd540ebac.el9.x8 8.8 MB/s | 3.3 MB 00:00 2024-04-13T13:09:29.554 INFO:teuthology.orchestra.run.smithi122.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-04-13T13:09:29.558 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:09:29.558 INFO:teuthology.orchestra.run.smithi122.stdout:Total 16 MB/s | 53 MB 00:03 2024-04-13T13:09:29.621 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:09:29.655 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:09:29.655 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:09:30.119 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:09:30.120 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:09:31.396 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-04-13T13:09:31.506 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:09:31.587 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-13T13:09:31.634 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-13T13:09:31.693 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-13T13:09:32.012 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-13T13:09:32.058 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 5/32 2024-04-13T13:09:32.149 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 5/32 2024-04-13T13:09:32.233 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:18.2.2-1187.gd540ebac.el9.x86_64 6/32 2024-04-13T13:09:32.276 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 7/32 2024-04-13T13:09:32.338 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 7/32 2024-04-13T13:09:32.367 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-13T13:09:32.411 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el9. 9/32 2024-04-13T13:09:32.468 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:18.2.2-1187.gd540ebac.el9.x86_64 10/32 2024-04-13T13:09:32.495 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:18.2.2-1187.gd540ebac.el9.x86_6 11/32 2024-04-13T13:09:32.560 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1187.gd540ebac.el9.x86_6 11/32 2024-04-13T13:09:32.668 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-13T13:09:32.703 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-13T13:09:32.758 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-13T13:09:32.791 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-13T13:09:32.818 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-13T13:09:32.854 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-13T13:09:32.887 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-13T13:09:33.188 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-13T13:09:33.209 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 20/32 2024-04-13T13:09:33.282 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 20/32 2024-04-13T13:09:33.320 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:18.2.2-1187.gd540ebac.el9.x86_64 21/32 2024-04-13T13:09:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-13T13:09:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-13T13:09:33.786 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:18.2.2-1187.gd540ebac.el9.x86_64 24/32 2024-04-13T13:09:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:18.2.2-1187.gd540ebac.el9.x86_64 24/32 2024-04-13T13:09:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:18.2.2-1187.gd540ebac.el9.x86_64 25/32 2024-04-13T13:09:33.956 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:09:33.956 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:09:33.956 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:09:33.956 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:09:33.957 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.958 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.959 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.960 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:09:33.961 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:09:33.962 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:09:33.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.965 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:09:33.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:09:33.972 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:18.2.2-1187.gd540ebac.el9.x8 26/32 2024-04-13T13:09:34.443 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 7.7 M 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 3.3 M 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 3.0 M 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:09:34.451 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 5.1 M 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 18 M 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 25 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 709 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 474 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-13T13:09:34.452 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 4.5 M 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 45 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 129 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 161 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 320 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 299 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 99 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:34.453 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:09:34.454 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:09:34.454 INFO:teuthology.orchestra.run.smithi083.stdout:Install 28 Packages 2024-04-13T13:09:34.454 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2024-04-13T13:09:34.454 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:34.454 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 53 M 2024-04-13T13:09:34.455 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:09:35.206 INFO:teuthology.orchestra.run.smithi083.stdout:(1/30): ceph-base-18.2.2-1187.gd540ebac.el9.x86 9.3 MB/s | 5.1 MB 00:00 2024-04-13T13:09:35.247 INFO:teuthology.orchestra.run.smithi083.stdout:(2/30): ceph-selinux-18.2.2-1187.gd540ebac.el9. 606 kB/s | 25 kB 00:00 2024-04-13T13:09:35.331 INFO:teuthology.orchestra.run.smithi083.stdout:(3/30): libcephfs2-18.2.2-1187.gd540ebac.el9.x8 8.3 MB/s | 709 kB 00:00 2024-04-13T13:09:35.381 INFO:teuthology.orchestra.run.smithi083.stdout:(4/30): libradosstriper1-18.2.2-1187.gd540ebac. 9.3 MB/s | 474 kB 00:00 2024-04-13T13:09:35.698 INFO:teuthology.orchestra.run.smithi083.stdout:(5/30): librgw2-18.2.2-1187.gd540ebac.el9.x86_6 14 MB/s | 4.5 MB 00:00 2024-04-13T13:09:35.757 INFO:teuthology.orchestra.run.smithi083.stdout:(6/30): python3-ceph-argparse-18.2.2-1187.gd540 773 kB/s | 45 kB 00:00 2024-04-13T13:09:35.824 INFO:teuthology.orchestra.run.smithi083.stdout:(7/30): python3-ceph-common-18.2.2-1187.gd540eb 1.9 MB/s | 129 kB 00:00 2024-04-13T13:09:35.882 INFO:teuthology.orchestra.run.smithi083.stdout:(8/30): python3-cephfs-18.2.2-1187.gd540ebac.el 2.7 MB/s | 161 kB 00:00 2024-04-13T13:09:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:(9/30): python3-rados-18.2.2-1187.gd540ebac.el9 4.2 MB/s | 320 kB 00:00 2024-04-13T13:09:36.016 INFO:teuthology.orchestra.run.smithi083.stdout:(10/30): python3-rbd-18.2.2-1187.gd540ebac.el9. 5.0 MB/s | 299 kB 00:00 2024-04-13T13:09:36.100 INFO:teuthology.orchestra.run.smithi083.stdout:(11/30): ceph-radosgw-18.2.2-1187.gd540ebac.el9 5.4 MB/s | 7.7 MB 00:01 2024-04-13T13:09:36.125 INFO:teuthology.orchestra.run.smithi083.stdout:(12/30): python3-rgw-18.2.2-1187.gd540ebac.el9. 915 kB/s | 99 kB 00:00 2024-04-13T13:09:36.135 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 27/32 2024-04-13T13:09:36.177 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 27/32 2024-04-13T13:09:36.551 INFO:teuthology.orchestra.run.smithi083.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 237 kB/s | 107 kB 00:00 2024-04-13T13:09:36.668 INFO:teuthology.orchestra.run.smithi083.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 388 kB/s | 45 kB 00:00 2024-04-13T13:09:36.701 INFO:teuthology.orchestra.run.smithi083.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 278 kB/s | 160 kB 00:00 2024-04-13T13:09:36.709 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 27/32 2024-04-13T13:09:36.747 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 28/32 2024-04-13T13:09:36.838 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 28/32 2024-04-13T13:09:36.838 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-13T13:09:36.838 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-13T13:09:36.838 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:36.881 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:18.2.2-1187.gd540ebac.el9.x86_64 29/32 2024-04-13T13:09:36.952 INFO:teuthology.orchestra.run.smithi083.stdout:(16/30): ceph-common-18.2.2-1187.gd540ebac.el9. 8.0 MB/s | 18 MB 00:02 2024-04-13T13:09:37.085 INFO:teuthology.orchestra.run.smithi083.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 762 kB/s | 292 kB 00:00 2024-04-13T13:09:37.261 INFO:teuthology.orchestra.run.smithi083.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 1.1 MB/s | 662 kB 00:00 2024-04-13T13:09:37.394 INFO:teuthology.orchestra.run.smithi083.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 96 kB/s | 42 kB 00:00 2024-04-13T13:09:37.536 INFO:teuthology.orchestra.run.smithi083.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 246 kB/s | 111 kB 00:00 2024-04-13T13:09:37.636 INFO:teuthology.orchestra.run.smithi083.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-04-13T13:09:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-04-13T13:09:37.829 INFO:teuthology.orchestra.run.smithi083.stdout:(23/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 544 kB/s | 309 kB 00:00 2024-04-13T13:09:37.870 INFO:teuthology.orchestra.run.smithi083.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2024-04-13T13:09:38.029 INFO:teuthology.orchestra.run.smithi083.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-13T13:09:38.246 INFO:teuthology.orchestra.run.smithi083.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-04-13T13:09:38.321 INFO:teuthology.orchestra.run.smithi083.stdout:(27/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.7 MB/s | 838 kB 00:00 2024-04-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi083.stdout:(28/30): librados2-18.2.2-1187.gd540ebac.el9.x8 12 MB/s | 3.3 MB 00:00 2024-04-13T13:09:38.772 INFO:teuthology.orchestra.run.smithi083.stdout:(29/30): librbd1-18.2.2-1187.gd540ebac.el9.x86_ 6.7 MB/s | 3.0 MB 00:00 2024-04-13T13:09:38.889 INFO:teuthology.orchestra.run.smithi083.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 1.8 MB/s | 1.6 MB 00:00 2024-04-13T13:09:38.891 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:09:38.892 INFO:teuthology.orchestra.run.smithi083.stdout:Total 12 MB/s | 53 MB 00:04 2024-04-13T13:09:38.964 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:09:38.998 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:09:38.998 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:09:39.445 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:09:39.445 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:09:40.844 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:09:40.930 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-13T13:09:40.960 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-13T13:09:41.019 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-13T13:09:41.389 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-13T13:09:41.417 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 5/32 2024-04-13T13:09:41.509 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 5/32 2024-04-13T13:09:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:18.2.2-1187.gd540ebac.el9.x86_64 6/32 2024-04-13T13:09:41.636 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 7/32 2024-04-13T13:09:41.698 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 7/32 2024-04-13T13:09:41.727 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-13T13:09:41.771 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el9. 9/32 2024-04-13T13:09:41.837 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:18.2.2-1187.gd540ebac.el9.x86_64 10/32 2024-04-13T13:09:41.863 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:18.2.2-1187.gd540ebac.el9.x86_6 11/32 2024-04-13T13:09:41.929 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1187.gd540ebac.el9.x86_6 11/32 2024-04-13T13:09:42.035 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-13T13:09:42.072 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-13T13:09:42.111 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-13T13:09:42.143 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-13T13:09:42.179 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-13T13:09:42.215 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-13T13:09:42.247 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-13T13:09:42.547 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-13T13:09:42.577 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 20/32 2024-04-13T13:09:42.659 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 20/32 2024-04-13T13:09:42.690 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:18.2.2-1187.gd540ebac.el9.x86_64 21/32 2024-04-13T13:09:42.726 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-13T13:09:43.119 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-13T13:09:43.146 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:18.2.2-1187.gd540ebac.el9.x86_64 24/32 2024-04-13T13:09:43.226 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:18.2.2-1187.gd540ebac.el9.x86_64 24/32 2024-04-13T13:09:43.302 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:18.2.2-1187.gd540ebac.el9.x86_64 25/32 2024-04-13T13:09:43.333 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:18.2.2-1187.gd540ebac.el9.x8 26/32 2024-04-13T13:09:45.396 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 27/32 2024-04-13T13:09:45.438 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 27/32 2024-04-13T13:09:45.998 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 27/32 2024-04-13T13:09:46.032 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 28/32 2024-04-13T13:09:46.123 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 28/32 2024-04-13T13:09:46.123 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-13T13:09:46.123 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-13T13:09:46.123 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:46.158 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:18.2.2-1187.gd540ebac.el9.x86_64 29/32 2024-04-13T13:09:46.357 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1187.gd540ebac.el9.x86_64 29/32 2024-04-13T13:09:46.357 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-04-13T13:09:46.358 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-04-13T13:09:46.358 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-04-13T13:09:46.358 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-04-13T13:09:46.358 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-04-13T13:09:46.358 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-04-13T13:09:46.358 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-04-13T13:09:46.358 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:47.099 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:18.2.2-1187.gd540ebac.el9.x86_64 30/32 2024-04-13T13:09:47.124 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1187.gd540ebac.el9.x86_64 30/32 2024-04-13T13:09:47.124 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:09:47.124 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-13T13:09:47.124 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T13:09:47.124 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T13:09:47.124 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-13T13:09:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-13T13:09:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-13T13:09:48.367 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-13T13:09:48.367 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 1/32 2024-04-13T13:09:48.367 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 2/32 2024-04-13T13:09:48.367 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:18.2.2-1187.gd540ebac.el9.x86_64 3/32 2024-04-13T13:09:48.367 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:18.2.2-1187.gd540ebac.el9.x86_64 4/32 2024-04-13T13:09:48.367 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 5/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:18.2.2-1187.gd540ebac.el9.x86_6 6/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:18.2.2-1187.gd540ebac.el9.x86_64 7/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el9. 8/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:18.2.2-1187.gd540ebac.el9.x8 9/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:18.2.2-1187.gd540ebac.el9.x86_64 10/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:18.2.2-1187.gd540ebac.el9.x86_64 11/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:18.2.2-1187.gd540ebac.el9.x86_64 12/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:18.2.2-1187.gd540ebac.el9.x86_64 13/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-13T13:09:48.368 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 29/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-13T13:09:48.369 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 31/32 2024-04-13T13:09:49.523 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-13T13:09:49.523 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:49.523 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-04-13T13:09:49.523 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.524 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-13T13:09:49.525 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.526 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.526 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:49.526 INFO:teuthology.orchestra.run.smithi122.stdout: re2-1:20211101-3.el9.x86_64 2024-04-13T13:09:49.526 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-13T13:09:49.526 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:49.526 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:09:49.893 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-04-13T13:09:50.481 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:27 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:09:50.769 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:09:50.770 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:09:50.771 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.772 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:09:50.773 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:09:50.774 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.775 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.776 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:09:50.777 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.778 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.778 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.778 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:09:50.842 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:09:50.843 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:09:50.843 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:09:50.843 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:09:50.843 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:09:50.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 40 M 2024-04-13T13:09:50.843 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 166 k 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-04-13T13:09:50.844 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:09:50.845 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 41 M 2024-04-13T13:09:50.845 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 164 M 2024-04-13T13:09:50.845 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:09:51.291 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 941 kB/s | 187 kB 00:00 2024-04-13T13:09:51.358 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): libcephsqlite-18.2.2-1187.gd540ebac.el9. 624 kB/s | 166 kB 00:00 2024-04-13T13:09:51.392 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.1 MB/s | 218 kB 00:00 2024-04-13T13:09:51.442 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-13T13:09:51.559 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 305 kB 00:00 2024-04-13T13:09:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): ceph-test-18.2.2-1187.gd540ebac.el9.x86_ 17 MB/s | 40 MB 00:02 2024-04-13T13:09:53.369 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:09:53.370 INFO:teuthology.orchestra.run.smithi122.stdout:Total 16 MB/s | 41 MB 00:02 2024-04-13T13:09:53.420 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:09:53.439 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:09:53.440 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:09:53.777 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:09:53.777 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:09:54.302 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:09:54.376 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-13T13:09:54.419 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-13T13:09:54.459 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-13T13:09:54.504 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-13T13:09:54.531 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:18.2.2-1187.gd540ebac.el9.x86_64 5/6 2024-04-13T13:09:55.705 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1187.gd540ebac.el9.x86_64 29/32 2024-04-13T13:09:55.705 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-04-13T13:09:55.705 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-04-13T13:09:55.705 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-04-13T13:09:55.705 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-04-13T13:09:55.705 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-04-13T13:09:55.705 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-04-13T13:09:55.706 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-04-13T13:09:55.706 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:56.456 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:18.2.2-1187.gd540ebac.el9.x86_64 30/32 2024-04-13T13:09:56.477 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1187.gd540ebac.el9.x86_64 30/32 2024-04-13T13:09:56.477 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:09:56.477 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-13T13:09:56.478 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T13:09:56.478 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T13:09:56.478 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:56.479 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-13T13:09:56.558 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-13T13:09:56.558 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 1/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 2/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:18.2.2-1187.gd540ebac.el9.x86_64 3/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:18.2.2-1187.gd540ebac.el9.x86_64 4/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 5/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:18.2.2-1187.gd540ebac.el9.x86_6 6/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:18.2.2-1187.gd540ebac.el9.x86_64 7/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el9. 8/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:18.2.2-1187.gd540ebac.el9.x8 9/32 2024-04-13T13:09:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:18.2.2-1187.gd540ebac.el9.x86_64 10/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:18.2.2-1187.gd540ebac.el9.x86_64 11/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:18.2.2-1187.gd540ebac.el9.x86_64 12/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:18.2.2-1187.gd540ebac.el9.x86_64 13/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-13T13:09:57.673 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-13T13:09:57.674 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-13T13:09:57.674 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-13T13:09:57.674 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-13T13:09:57.674 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-13T13:09:57.674 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-13T13:09:57.675 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 29/32 2024-04-13T13:09:57.675 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-13T13:09:57.675 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 31/32 2024-04-13T13:09:58.244 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1187.gd540ebac.el9.x86_64 5/6 2024-04-13T13:09:58.313 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:18.2.2-1187.gd540ebac.el9.x86_64 6/6 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.697 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-13T13:09:58.698 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-3.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:58.699 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:09:59.042 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2024-04-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:28 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.910 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.911 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:09:59.912 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.913 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:09:59.914 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:09:59.915 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.916 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.917 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.917 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.917 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:09:59.917 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.917 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.917 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.917 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.918 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.919 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:09:59.988 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 40 M 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 166 k 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-13T13:09:59.989 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:59.990 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:09:59.990 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:09:59.990 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-04-13T13:09:59.990 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:09:59.990 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 41 M 2024-04-13T13:09:59.990 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 164 M 2024-04-13T13:09:59.990 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:10:00.388 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:18.2.2-1187.gd540ebac.el9.x86_64 6/6 2024-04-13T13:10:00.388 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:18.2.2-1187.gd540ebac.el9.x86_64 1/6 2024-04-13T13:10:00.388 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:18.2.2-1187.gd540ebac.el9.x86_64 2/6 2024-04-13T13:10:00.388 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-04-13T13:10:00.388 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-04-13T13:10:00.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-04-13T13:10:00.602 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 926 kB/s | 187 kB 00:00 2024-04-13T13:10:00.702 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.1 MB/s | 218 kB 00:00 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-16.el9.x86_64 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:00.792 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:00.861 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): libcephsqlite-18.2.2-1187.gd540ebac.el9. 360 kB/s | 166 kB 00:00 2024-04-13T13:10:01.107 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-04-13T13:10:01.586 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 88 kB/s | 64 kB 00:00 2024-04-13T13:10:01.712 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:38 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:01.769 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): socat-1.7.4.1-5.el9.x86_64.rpm 286 kB/s | 305 kB 00:01 2024-04-13T13:10:01.999 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:01.999 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:01.999 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:01.999 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:02.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:02.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:02.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:02.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:02.007 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.008 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.008 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.008 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.008 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:02.008 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.008 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.008 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.008 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:02.078 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:02.080 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:02.080 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:10:02.080 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:02.080 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 6.5 k 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 2.1 M 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 1.5 M 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 246 k 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 4.7 M 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 17 M 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-13T13:10:02.081 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-13T13:10:02.082 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:02.083 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:02.084 INFO:teuthology.orchestra.run.smithi122.stdout:Install 33 Packages 2024-04-13T13:10:02.084 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:02.084 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 29 M 2024-04-13T13:10:02.084 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 104 M 2024-04-13T13:10:02.084 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:02.603 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): ceph-test-18.2.2-1187.gd540ebac.el9.x86_ 18 MB/s | 40 MB 00:02 2024-04-13T13:10:02.605 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:02.605 INFO:teuthology.orchestra.run.smithi083.stdout:Total 16 MB/s | 41 MB 00:02 2024-04-13T13:10:02.658 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:10:02.677 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:10:02.678 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:10:03.009 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:10:03.009 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:10:03.056 INFO:teuthology.orchestra.run.smithi122.stdout:(1/33): ceph-18.2.2-1187.gd540ebac.el9.x86_64.r 26 kB/s | 6.5 kB 00:00 2024-04-13T13:10:03.156 INFO:teuthology.orchestra.run.smithi122.stdout:(2/33): ceph-mgr-18.2.2-1187.gd540ebac.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-04-13T13:10:03.544 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:10:03.645 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-13T13:10:03.687 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-13T13:10:03.728 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-13T13:10:03.778 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-13T13:10:03.806 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:18.2.2-1187.gd540ebac.el9.x86_64 5/6 2024-04-13T13:10:03.956 INFO:teuthology.orchestra.run.smithi122.stdout:(3/33): ceph-osd-18.2.2-1187.gd540ebac.el9.x86_ 21 MB/s | 17 MB 00:00 2024-04-13T13:10:04.006 INFO:teuthology.orchestra.run.smithi122.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1187.gd540 4.8 MB/s | 246 kB 00:00 2024-04-13T13:10:04.365 INFO:teuthology.orchestra.run.smithi122.stdout:(5/33): ceph-mds-18.2.2-1187.gd540ebac.el9.x86_ 1.3 MB/s | 2.1 MB 00:01 2024-04-13T13:10:04.657 INFO:teuthology.orchestra.run.smithi122.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 389 kB/s | 253 kB 00:00 2024-04-13T13:10:04.782 INFO:teuthology.orchestra.run.smithi122.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 852 kB/s | 106 kB 00:00 2024-04-13T13:10:04.949 INFO:teuthology.orchestra.run.smithi122.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 810 kB/s | 135 kB 00:00 2024-04-13T13:10:05.008 INFO:teuthology.orchestra.run.smithi122.stdout:(9/33): ceph-mon-18.2.2-1187.gd540ebac.el9.x86_ 2.4 MB/s | 4.7 MB 00:01 2024-04-13T13:10:05.075 INFO:teuthology.orchestra.run.smithi122.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 997 kB/s | 125 kB 00:00 2024-04-13T13:10:05.200 INFO:teuthology.orchestra.run.smithi122.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-13T13:10:05.242 INFO:teuthology.orchestra.run.smithi122.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 836 kB/s | 35 kB 00:00 2024-04-13T13:10:05.284 INFO:teuthology.orchestra.run.smithi122.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-13T13:10:05.326 INFO:teuthology.orchestra.run.smithi122.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 999 kB/s | 42 kB 00:00 2024-04-13T13:10:05.359 INFO:teuthology.orchestra.run.smithi122.stdout:(15/33): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:00 2024-04-13T13:10:05.602 INFO:teuthology.orchestra.run.smithi122.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 158 kB/s | 43 kB 00:00 2024-04-13T13:10:05.626 INFO:teuthology.orchestra.run.smithi122.stdout:(17/33): python3-urllib3-1.26.5-5.el9.noarch.rp 347 kB/s | 215 kB 00:00 2024-04-13T13:10:05.719 INFO:teuthology.orchestra.run.smithi122.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 478 kB/s | 172 kB 00:00 2024-04-13T13:10:05.785 INFO:teuthology.orchestra.run.smithi122.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 266 kB/s | 18 kB 00:00 2024-04-13T13:10:05.802 INFO:teuthology.orchestra.run.smithi122.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2024-04-13T13:10:05.828 INFO:teuthology.orchestra.run.smithi122.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-04-13T13:10:05.844 INFO:teuthology.orchestra.run.smithi122.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2024-04-13T13:10:05.878 INFO:teuthology.orchestra.run.smithi122.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-04-13T13:10:05.895 INFO:teuthology.orchestra.run.smithi122.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-04-13T13:10:05.921 INFO:teuthology.orchestra.run.smithi122.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 611 kB/s | 46 kB 00:00 2024-04-13T13:10:05.988 INFO:teuthology.orchestra.run.smithi122.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-13T13:10:06.013 INFO:teuthology.orchestra.run.smithi122.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-04-13T13:10:06.038 INFO:teuthology.orchestra.run.smithi122.stdout:(28/33): python3-more-itertools-8.12.0-2.el9.no 494 kB/s | 79 kB 00:00 2024-04-13T13:10:06.063 INFO:teuthology.orchestra.run.smithi122.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-13T13:10:06.097 INFO:teuthology.orchestra.run.smithi122.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-04-13T13:10:06.164 INFO:teuthology.orchestra.run.smithi122.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-04-13T13:10:06.189 INFO:teuthology.orchestra.run.smithi122.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-04-13T13:10:06.214 INFO:teuthology.orchestra.run.smithi122.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-04-13T13:10:06.219 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:06.219 INFO:teuthology.orchestra.run.smithi122.stdout:Total 7.0 MB/s | 29 MB 00:04 2024-04-13T13:10:06.339 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:06.375 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:06.376 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:06.751 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:06.751 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:07.469 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1187.gd540ebac.el9.x86_64 5/6 2024-04-13T13:10:07.547 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:18.2.2-1187.gd540ebac.el9.x86_64 6/6 2024-04-13T13:10:07.991 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:08.051 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-13T13:10:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-13T13:10:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-13T13:10:08.206 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-13T13:10:08.294 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-13T13:10:08.336 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-13T13:10:08.370 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-13T13:10:08.400 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-13T13:10:08.426 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-13T13:10:08.459 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-13T13:10:08.500 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-13T13:10:08.543 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-13T13:10:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-13T13:10:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-13T13:10:08.689 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-13T13:10:08.826 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-13T13:10:08.906 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-13T13:10:08.969 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-13T13:10:09.007 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-13T13:10:09.056 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-13T13:10:09.129 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-13T13:10:09.316 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-13T13:10:09.370 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-13T13:10:09.447 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-13T13:10:09.537 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-13T13:10:09.585 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-13T13:10:09.700 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-13T13:10:09.753 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:18.2.2-1187.gd540ebac.el9.x86_64 6/6 2024-04-13T13:10:09.753 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:18.2.2-1187.gd540ebac.el9.x86_64 1/6 2024-04-13T13:10:09.753 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:18.2.2-1187.gd540ebac.el9.x86_64 2/6 2024-04-13T13:10:09.753 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-04-13T13:10:09.754 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-04-13T13:10:09.754 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-04-13T13:10:09.851 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el9. 28/33 2024-04-13T13:10:09.879 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 29/33 2024-04-13T13:10:09.905 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 29/33 2024-04-13T13:10:09.905 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:09.905 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-13T13:10:09.905 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T13:10:09.905 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T13:10:09.905 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:10.130 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-04-13T13:10:10.130 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:10.130 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:10:10.130 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:10.130 INFO:teuthology.orchestra.run.smithi083.stdout: jq-1.6-16.el9.x86_64 2024-04-13T13:10:10.131 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:10.131 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-13T13:10:10.131 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-13T13:10:10.131 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-13T13:10:10.131 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:10.131 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:10.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2024-04-13T13:10:11.004 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:40 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:11.290 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:11.290 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:11.290 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:11.290 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:11.291 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.292 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.293 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.294 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:11.295 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.296 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.297 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.298 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.299 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:11.356 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:18.2.2-1187.gd540ebac.el9.x86_64 30/33 2024-04-13T13:10:11.360 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:11.362 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 6.5 k 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 2.1 M 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 1.5 M 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 246 k 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 4.7 M 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 17 M 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-13T13:10:11.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-13T13:10:11.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:11.365 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:10:11.366 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:11.366 INFO:teuthology.orchestra.run.smithi083.stdout:Install 33 Packages 2024-04-13T13:10:11.366 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:11.366 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 29 M 2024-04-13T13:10:11.366 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 104 M 2024-04-13T13:10:11.366 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:10:11.383 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:18.2.2-1187.gd540ebac.el9.x86_64 30/33 2024-04-13T13:10:11.383 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:11.383 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-13T13:10:11.383 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T13:10:11.383 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T13:10:11.383 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:11.831 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:18.2.2-1187.gd540ebac.el9.x86_64 31/33 2024-04-13T13:10:11.855 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:18.2.2-1187.gd540ebac.el9.x86_64 31/33 2024-04-13T13:10:11.855 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:11.856 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-13T13:10:11.856 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T13:10:11.856 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T13:10:11.856 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:12.040 INFO:teuthology.orchestra.run.smithi083.stdout:(1/33): ceph-18.2.2-1187.gd540ebac.el9.x86_64.r 30 kB/s | 6.5 kB 00:00 2024-04-13T13:10:12.057 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:18.2.2-1187.gd540ebac.el9.x86_64 32/33 2024-04-13T13:10:12.080 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:18.2.2-1187.gd540ebac.el9.x86_64 32/33 2024-04-13T13:10:12.080 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:12.080 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-13T13:10:12.081 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T13:10:12.081 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T13:10:12.081 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:12.147 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:18.2.2-1187.gd540ebac.el9.x86_64 33/33 2024-04-13T13:10:12.749 INFO:teuthology.orchestra.run.smithi083.stdout:(2/33): ceph-mgr-18.2.2-1187.gd540ebac.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2024-04-13T13:10:12.865 INFO:teuthology.orchestra.run.smithi083.stdout:(3/33): ceph-mds-18.2.2-1187.gd540ebac.el9.x86_ 2.0 MB/s | 2.1 MB 00:01 2024-04-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi083.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1187.gd540 2.2 MB/s | 246 kB 00:00 2024-04-13T13:10:13.283 INFO:teuthology.orchestra.run.smithi083.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 821 kB/s | 253 kB 00:00 2024-04-13T13:10:13.391 INFO:teuthology.orchestra.run.smithi083.stdout:(6/33): ceph-mon-18.2.2-1187.gd540ebac.el9.x86_ 3.5 MB/s | 4.7 MB 00:01 2024-04-13T13:10:13.475 INFO:teuthology.orchestra.run.smithi083.stdout:(7/33): python3-cryptography-36.0.1-4.el9.x86_6 6.4 MB/s | 1.2 MB 00:00 2024-04-13T13:10:13.542 INFO:teuthology.orchestra.run.smithi083.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-04-13T13:10:13.609 INFO:teuthology.orchestra.run.smithi083.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-04-13T13:10:13.634 INFO:teuthology.orchestra.run.smithi083.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-04-13T13:10:13.676 INFO:teuthology.orchestra.run.smithi083.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:18.2.2-1187.gd540ebac.el9.x86_64 33/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:18.2.2-1187.gd540ebac.el9.x86_64 1/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:18.2.2-1187.gd540ebac.el9.x86_64 2/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 3/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:18.2.2-1187.gd540ebac.el9.x86_64 4/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:18.2.2-1187.gd540ebac.el9.x86_64 5/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el9. 6/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-13T13:10:13.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-13T13:10:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-13T13:10:13.927 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-13T13:10:13.927 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-13T13:10:13.927 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-13T13:10:13.927 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-13T13:10:13.928 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-13T13:10:13.928 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-13T13:10:13.928 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-13T13:10:13.928 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-13T13:10:14.293 INFO:teuthology.orchestra.run.smithi083.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 56 kB/s | 35 kB 00:00 2024-04-13T13:10:14.554 INFO:teuthology.orchestra.run.smithi083.stdout:(13/33): ceph-osd-18.2.2-1187.gd540ebac.el9.x86 9.2 MB/s | 17 MB 00:01 2024-04-13T13:10:14.577 INFO:teuthology.orchestra.run.smithi083.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 183 kB/s | 172 kB 00:00 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:14.997 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-13T13:10:14.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-13T13:10:14.999 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-13T13:10:15.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-13T13:10:15.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-13T13:10:15.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-13T13:10:15.000 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:15.000 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:15.052 INFO:teuthology.orchestra.run.smithi083.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 67 kB/s | 51 kB 00:00 2024-04-13T13:10:15.111 INFO:teuthology.orchestra.run.smithi083.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 81 kB/s | 43 kB 00:00 2024-04-13T13:10:15.186 INFO:teuthology.orchestra.run.smithi083.stdout:(17/33): python3-toml-0.10.2-6.el9.noarch.rpm 66 kB/s | 42 kB 00:00 2024-04-13T13:10:15.368 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-04-13T13:10:15.561 INFO:teuthology.orchestra.run.smithi083.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 28 kB/s | 11 kB 00:00 2024-04-13T13:10:15.636 INFO:teuthology.orchestra.run.smithi083.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 682 kB/s | 358 kB 00:00 2024-04-13T13:10:15.662 INFO:teuthology.orchestra.run.smithi083.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-04-13T13:10:15.754 INFO:teuthology.orchestra.run.smithi083.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-04-13T13:10:15.788 INFO:teuthology.orchestra.run.smithi083.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 154 kB/s | 19 kB 00:00 2024-04-13T13:10:15.813 INFO:teuthology.orchestra.run.smithi083.stdout:(23/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 226 kB/s | 172 kB 00:00 2024-04-13T13:10:15.854 INFO:teuthology.orchestra.run.smithi083.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-04-13T13:10:15.961 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:52 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:15.980 INFO:teuthology.orchestra.run.smithi083.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 242 kB/s | 46 kB 00:00 2024-04-13T13:10:16.005 INFO:teuthology.orchestra.run.smithi083.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 410 kB/s | 79 kB 00:00 2024-04-13T13:10:16.080 INFO:teuthology.orchestra.run.smithi083.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-04-13T13:10:16.180 INFO:teuthology.orchestra.run.smithi083.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-04-13T13:10:16.206 INFO:teuthology.orchestra.run.smithi083.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 447 kB/s | 90 kB 00:00 2024-04-13T13:10:16.250 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:16.250 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:16.250 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:16.250 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.250 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:16.251 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:16.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:16.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:16.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:16.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.257 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:16.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.258 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:16.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.260 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:16.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:16.260 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:10:16.273 INFO:teuthology.orchestra.run.smithi083.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 651 kB/s | 272 kB 00:00 2024-04-13T13:10:16.319 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:16.320 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:10:16.320 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:16.373 INFO:teuthology.orchestra.run.smithi083.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-04-13T13:10:16.390 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-04-13T13:10:16.557 INFO:teuthology.orchestra.run.smithi083.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 611 kB/s | 230 kB 00:00 2024-04-13T13:10:16.832 INFO:teuthology.orchestra.run.smithi083.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 683 kB/s | 427 kB 00:00 2024-04-13T13:10:16.838 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:16.838 INFO:teuthology.orchestra.run.smithi083.stdout:Total 5.3 MB/s | 29 MB 00:05 2024-04-13T13:10:16.958 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:10:16.978 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:53 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:16.995 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:10:16.995 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:10:17.266 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:17.266 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:17.267 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.268 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.269 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:17.270 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:17.271 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.272 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.273 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.274 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.275 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:17.334 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:17.335 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:17.335 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:10:17.335 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:17.335 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:17.335 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 224 k 2024-04-13T13:10:17.335 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:17.335 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:17.335 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:17.336 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-13T13:10:17.336 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:17.336 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 224 k 2024-04-13T13:10:17.336 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 216 k 2024-04-13T13:10:17.336 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:17.361 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:10:17.361 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:10:17.832 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-18.2.2-1187.gd540ebac.el9.noarch.rpm 452 kB/s | 224 kB 00:00 2024-04-13T13:10:17.832 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:17.833 INFO:teuthology.orchestra.run.smithi122.stdout:Total 451 kB/s | 224 kB 00:00 2024-04-13T13:10:17.833 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:17.838 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:17.838 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:17.840 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:17.841 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:17.913 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:18.286 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/1 2024-04-13T13:10:18.386 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/1 2024-04-13T13:10:18.611 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:10:18.668 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-13T13:10:18.701 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-13T13:10:18.733 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-13T13:10:18.760 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/1 2024-04-13T13:10:18.825 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-13T13:10:18.903 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-13T13:10:18.944 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-13T13:10:18.988 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-13T13:10:19.017 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-13T13:10:19.029 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/1 2024-04-13T13:10:19.029 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:19.029 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:19.029 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:19.030 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:19.030 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:19.052 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-13T13:10:19.085 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-13T13:10:19.135 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-13T13:10:19.169 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-13T13:10:19.196 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-13T13:10:19.227 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-04-13T13:10:19.234 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-13T13:10:19.307 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-13T13:10:19.452 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-13T13:10:19.533 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-13T13:10:19.587 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-13T13:10:19.625 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-13T13:10:19.683 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-13T13:10:19.744 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-13T13:10:19.840 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:56 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:19.942 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-13T13:10:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-13T13:10:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-13T13:10:20.129 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:20.129 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:20.129 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:20.129 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.130 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:20.131 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.132 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:20.133 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:20.134 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:20.135 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.136 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:20.137 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.138 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.138 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.138 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.138 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:20.138 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.138 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.138 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.138 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:20.154 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-13T13:10:20.187 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-13T13:10:20.198 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:20.198 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:20.198 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-13T13:10:20.198 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:20.198 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:20.199 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 143 k 2024-04-13T13:10:20.199 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:20.199 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:20.199 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:20.199 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-13T13:10:20.199 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:20.199 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 143 k 2024-04-13T13:10:20.200 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 395 k 2024-04-13T13:10:20.200 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:20.303 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-13T13:10:20.414 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-18.2.2-1187.gd540eb 665 kB/s | 143 kB 00:00 2024-04-13T13:10:20.415 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:20.415 INFO:teuthology.orchestra.run.smithi122.stdout:Total 660 kB/s | 143 kB 00:00 2024-04-13T13:10:20.415 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:20.423 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:20.423 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:20.454 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el9. 28/33 2024-04-13T13:10:20.476 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:20.476 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:20.481 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 29/33 2024-04-13T13:10:20.505 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 29/33 2024-04-13T13:10:20.505 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:20.506 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-13T13:10:20.506 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T13:10:20.506 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T13:10:20.506 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:20.622 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:20.699 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-13T13:10:20.723 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-13T13:10:20.724 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:20.724 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-13T13:10:20.724 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-13T13:10:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:21.752 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:21.753 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:21.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-04-13T13:10:21.984 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:18.2.2-1187.gd540ebac.el9.x86_64 30/33 2024-04-13T13:10:22.011 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:18.2.2-1187.gd540ebac.el9.x86_64 30/33 2024-04-13T13:10:22.011 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:22.011 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-13T13:10:22.011 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T13:10:22.011 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T13:10:22.011 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:22.442 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:18.2.2-1187.gd540ebac.el9.x86_64 31/33 2024-04-13T13:10:22.465 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:18.2.2-1187.gd540ebac.el9.x86_64 31/33 2024-04-13T13:10:22.465 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:22.465 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-13T13:10:22.465 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T13:10:22.466 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T13:10:22.466 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:22.582 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:59 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:22.668 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:18.2.2-1187.gd540ebac.el9.x86_64 32/33 2024-04-13T13:10:22.690 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:18.2.2-1187.gd540ebac.el9.x86_64 32/33 2024-04-13T13:10:22.691 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:22.691 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-13T13:10:22.691 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T13:10:22.691 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T13:10:22.691 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:22.749 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:18.2.2-1187.gd540ebac.el9.x86_64 33/33 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.869 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.870 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:22.871 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:22.872 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:22.873 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:22.874 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.875 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.877 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:22.878 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.878 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.878 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.878 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:22.878 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:10:22.935 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:22.936 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:10:22.937 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:23.002 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-04-13T13:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:00 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:23.883 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.884 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.885 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.886 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:23.887 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:23.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.892 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:23.951 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 3.5 M 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 24 k 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 15 k 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:23.952 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:23.953 INFO:teuthology.orchestra.run.smithi122.stdout:Install 5 Packages 2024-04-13T13:10:23.953 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:23.953 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.8 M 2024-04-13T13:10:23.953 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 87 M 2024-04-13T13:10:23.953 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:24.402 INFO:teuthology.orchestra.run.smithi122.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1187.gd540 71 kB/s | 15 kB 00:00 2024-04-13T13:10:24.436 INFO:teuthology.orchestra.run.smithi122.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1187.gd54 100 kB/s | 24 kB 00:00 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:18.2.2-1187.gd540ebac.el9.x86_64 33/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:18.2.2-1187.gd540ebac.el9.x86_64 1/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:18.2.2-1187.gd540ebac.el9.x86_64 2/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 3/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:18.2.2-1187.gd540ebac.el9.x86_64 4/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:18.2.2-1187.gd540ebac.el9.x86_64 5/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el9. 6/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-13T13:10:24.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-13T13:10:24.596 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-13T13:10:24.597 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-13T13:10:24.597 INFO:teuthology.orchestra.run.smithi122.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 160 kB/s | 31 kB 00:00 2024-04-13T13:10:24.598 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-13T13:10:24.598 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-13T13:10:24.598 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-13T13:10:24.598 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-13T13:10:24.598 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-13T13:10:24.598 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-13T13:10:24.598 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-13T13:10:24.711 INFO:teuthology.orchestra.run.smithi122.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 685 kB/s | 188 kB 00:00 2024-04-13T13:10:24.911 INFO:teuthology.orchestra.run.smithi122.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1187.gd540ebac 4.9 MB/s | 3.5 MB 00:00 2024-04-13T13:10:24.913 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:24.913 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.9 MB/s | 3.8 MB 00:00 2024-04-13T13:10:24.913 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:24.920 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:24.920 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:24.993 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:24.993 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:25.244 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:25.324 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-04-13T13:10:25.357 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-04-13T13:10:25.386 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el9.n 3/5 2024-04-13T13:10:25.531 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-13T13:10:25.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:25.531 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:10:25.531 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:25.531 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:25.531 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:25.531 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:25.531 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-13T13:10:25.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-13T13:10:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:25.534 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:25.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2024-04-13T13:10:26.428 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el9. 4/5 2024-04-13T13:10:26.457 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el9.noarc 5/5 2024-04-13T13:10:26.462 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:55 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:26.748 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:26.749 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:26.750 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.751 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:26.752 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.753 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:26.753 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.753 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:26.753 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:26.759 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:10:26.815 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:26.816 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:10:26.816 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:26.881 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2024-04-13T13:10:27.464 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:56 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el9.noarc 5/5 2024-04-13T13:10:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el9. 1/5 2024-04-13T13:10:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el9.noarc 2/5 2024-04-13T13:10:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el9.n 3/5 2024-04-13T13:10:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-04-13T13:10:27.753 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:27.754 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:27.755 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.756 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:27.757 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.758 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.759 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.760 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.761 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.762 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.762 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:27.762 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.762 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.762 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.762 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:27.821 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:27.822 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:10:27.822 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:27.822 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:10:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 224 k 2024-04-13T13:10:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:27.822 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:10:27.822 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:27.823 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-04-13T13:10:27.823 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:27.823 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 224 k 2024-04-13T13:10:27.823 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 216 k 2024-04-13T13:10:27.823 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:10:27.932 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-04-13T13:10:27.932 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:27.932 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:27.932 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:27.932 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:27.933 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:27.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-13T13:10:27.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-13T13:10:27.933 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:27.933 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:28.166 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-13T13:10:28.216 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-18.2.2-1187.gd540ebac.el9.noarch.rpm 570 kB/s | 224 kB 00:00 2024-04-13T13:10:28.217 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:28.217 INFO:teuthology.orchestra.run.smithi083.stdout:Total 568 kB/s | 224 kB 00:00 2024-04-13T13:10:28.217 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:10:28.222 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:10:28.222 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:10:28.224 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:10:28.225 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:10:28.299 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:10:28.697 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/1 2024-04-13T13:10:28.762 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:05 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:28.790 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/1 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.052 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.054 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.055 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:29.056 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:29.057 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.058 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.059 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.060 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:29.121 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:29.122 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 7.4 M 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-13T13:10:29.123 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-13T13:10:29.124 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-13T13:10:29.124 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-13T13:10:29.124 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-13T13:10:29.124 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:29.124 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:29.124 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-13T13:10:29.124 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-04-13T13:10:29.124 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:29.125 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 38 M 2024-04-13T13:10:29.125 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 209 M 2024-04-13T13:10:29.125 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:29.150 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/1 2024-04-13T13:10:29.438 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/1 2024-04-13T13:10:29.438 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:29.438 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:10:29.438 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:29.438 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:29.438 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:29.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2024-04-13T13:10:29.985 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 933 kB/s | 195 kB 00:00 2024-04-13T13:10:30.043 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-04-13T13:10:30.194 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 157 kB/s | 33 kB 00:00 2024-04-13T13:10:30.235 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:59 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:30.252 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 313 kB/s | 18 kB 00:00 2024-04-13T13:10:30.319 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 35 kB 00:00 2024-04-13T13:10:30.403 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.2-11 12 MB/s | 7.4 MB 00:00 2024-04-13T13:10:30.470 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-04-13T13:10:30.527 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:30.528 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.529 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.530 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.531 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.532 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.533 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.534 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.535 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.536 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.536 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:30.536 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.536 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.536 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.536 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:30.536 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-04-13T13:10:30.596 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:30.596 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:30.596 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-04-13T13:10:30.597 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:30.597 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:10:30.597 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 143 k 2024-04-13T13:10:30.597 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:30.597 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:10:30.597 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:30.597 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-04-13T13:10:30.597 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:30.598 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 143 k 2024-04-13T13:10:30.598 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 395 k 2024-04-13T13:10:30.598 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:10:30.671 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-04-13T13:10:30.696 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.7 MB/s | 250 kB 00:00 2024-04-13T13:10:30.826 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-18.2.2-1187.gd540eb 626 kB/s | 143 kB 00:00 2024-04-13T13:10:30.826 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:30.827 INFO:teuthology.orchestra.run.smithi083.stdout:Total 622 kB/s | 143 kB 00:00 2024-04-13T13:10:30.827 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:10:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:10:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:10:30.886 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:10:30.887 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:10:31.025 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:10:31.093 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-13T13:10:31.118 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-13T13:10:31.118 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:31.118 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-13T13:10:31.118 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:31.154 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-04-13T13:10:31.471 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.1 MB/s | 5.1 MB 00:01 2024-04-13T13:10:31.474 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:31.475 INFO:teuthology.orchestra.run.smithi122.stdout:Total 16 MB/s | 38 MB 00:02 2024-04-13T13:10:31.823 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:31.851 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:31.851 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:32.128 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-13T13:10:32.129 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:32.129 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:10:32.129 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:10:32.129 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:32.129 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:32.343 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:32.344 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:32.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2024-04-13T13:10:32.976 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:01 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:33.095 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:33.177 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-13T13:10:33.211 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-13T13:10:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:33.265 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:33.265 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:33.265 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.266 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:33.267 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:33.268 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:33.269 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:33.270 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.271 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:33.275 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:10:33.313 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-13T13:10:33.332 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:33.333 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:10:33.333 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-13T13:10:33.393 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2024-04-13T13:10:33.525 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-13T13:10:33.553 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-13T13:10:33.634 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-13T13:10:33.982 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:02 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:34.271 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.272 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:34.273 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:34.274 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:34.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.281 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:34.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:34.290 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-13T13:10:34.339 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 3.5 M 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 24 k 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 15 k 2024-04-13T13:10:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-13T13:10:34.341 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-13T13:10:34.341 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:34.341 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:10:34.341 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:34.341 INFO:teuthology.orchestra.run.smithi083.stdout:Install 5 Packages 2024-04-13T13:10:34.341 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:34.341 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.8 M 2024-04-13T13:10:34.342 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 87 M 2024-04-13T13:10:34.342 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:10:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-13T13:10:34.622 INFO:teuthology.orchestra.run.smithi083.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1187.gd54 133 kB/s | 24 kB 00:00 2024-04-13T13:10:34.638 INFO:teuthology.orchestra.run.smithi083.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1187.gd540 74 kB/s | 15 kB 00:00 2024-04-13T13:10:34.897 INFO:teuthology.orchestra.run.smithi083.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 112 kB/s | 31 kB 00:00 2024-04-13T13:10:35.056 INFO:teuthology.orchestra.run.smithi083.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 452 kB/s | 188 kB 00:00 2024-04-13T13:10:35.114 INFO:teuthology.orchestra.run.smithi083.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1187.gd540ebac 5.2 MB/s | 3.5 MB 00:00 2024-04-13T13:10:35.118 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:35.118 INFO:teuthology.orchestra.run.smithi083.stdout:Total 4.9 MB/s | 3.8 MB 00:00 2024-04-13T13:10:35.118 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:10:35.125 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:10:35.125 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:10:35.199 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:10:35.200 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:10:35.462 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:10:35.536 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-04-13T13:10:35.586 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-04-13T13:10:35.615 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el9.n 3/5 2024-04-13T13:10:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-13T13:10:36.662 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el9. 4/5 2024-04-13T13:10:36.664 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-13T13:10:36.694 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540e 12/12 2024-04-13T13:10:36.702 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el9.noarc 5/5 2024-04-13T13:10:37.916 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el9.noarc 5/5 2024-04-13T13:10:37.916 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el9. 1/5 2024-04-13T13:10:37.916 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el9.noarc 2/5 2024-04-13T13:10:37.916 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el9.n 3/5 2024-04-13T13:10:37.916 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-13T13:10:38.225 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:38.226 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:38.433 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-13T13:10:39.027 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:08 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:39.261 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540e 12/12 2024-04-13T13:10:39.261 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540e 1/12 2024-04-13T13:10:39.261 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-13T13:10:39.262 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-13T13:10:39.317 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:39.317 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:39.317 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:39.317 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.318 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:39.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:39.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.325 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:39.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:39.386 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:39.387 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-04-13T13:10:39.387 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:10:39.387 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-04-13T13:10:39.387 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 7.4 M 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-13T13:10:39.388 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:39.389 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:10:39.389 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-04-13T13:10:39.389 INFO:teuthology.orchestra.run.smithi083.stdout:Install 12 Packages 2024-04-13T13:10:39.389 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:39.389 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 38 M 2024-04-13T13:10:39.389 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 209 M 2024-04-13T13:10:39.390 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:10:39.762 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-13T13:10:39.762 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:39.762 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:39.762 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:39.762 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-13T13:10:39.762 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-13T13:10:39.762 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-13T13:10:39.762 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:39.763 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:39.847 INFO:teuthology.orchestra.run.smithi083.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-04-13T13:10:39.963 INFO:teuthology.orchestra.run.smithi083.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-04-13T13:10:40.014 INFO:teuthology.orchestra.run.smithi083.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 197 kB/s | 33 kB 00:00 2024-04-13T13:10:40.072 INFO:teuthology.orchestra.run.smithi083.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-04-13T13:10:40.131 INFO:teuthology.orchestra.run.smithi083.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 605 kB/s | 35 kB 00:00 2024-04-13T13:10:40.186 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-04-13T13:10:40.573 INFO:teuthology.orchestra.run.smithi083.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-04-13T13:10:40.632 INFO:teuthology.orchestra.run.smithi083.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-04-13T13:10:40.807 INFO:teuthology.orchestra.run.smithi083.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.0 MB/s | 4.7 MB 00:00 2024-04-13T13:10:40.809 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:17 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:40.866 INFO:teuthology.orchestra.run.smithi083.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-04-13T13:10:41.099 INFO:teuthology.orchestra.run.smithi083.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-13T13:10:41.117 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:41.117 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:41.117 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:41.118 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.119 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.120 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.121 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.123 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.124 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.125 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.126 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:41.187 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:41.189 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:41.189 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:10:41.189 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:41.189 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:41.189 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 49 k 2024-04-13T13:10:41.189 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:10:41.189 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-13T13:10:41.189 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:41.190 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-04-13T13:10:41.191 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:41.191 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.2 M 2024-04-13T13:10:41.191 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-04-13T13:10:41.191 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:41.543 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 217 kB/s | 19 kB 00:00 2024-04-13T13:10:41.560 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 243 kB/s | 26 kB 00:00 2024-04-13T13:10:41.574 INFO:teuthology.orchestra.run.smithi083.stdout:(11/12): ceph-mgr-diskprediction-local-18.2.2-1 4.0 MB/s | 7.4 MB 00:01 2024-04-13T13:10:41.644 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-04-13T13:10:41.661 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): ceph-mgr-rook-18.2.2-1187.gd540ebac.el9 239 kB/s | 49 kB 00:00 2024-04-13T13:10:41.686 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-13T13:10:41.711 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.0 MB/s | 279 kB 00:00 2024-04-13T13:10:41.753 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 803 kB/s | 54 kB 00:00 2024-04-13T13:10:41.891 INFO:teuthology.orchestra.run.smithi083.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:01 2024-04-13T13:10:41.893 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:41.893 INFO:teuthology.orchestra.run.smithi083.stdout:Total 15 MB/s | 38 MB 00:02 2024-04-13T13:10:41.970 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-04-13T13:10:41.996 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-04-13T13:10:42.146 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-04-13T13:10:42.171 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 547 kB/s | 203 kB 00:00 2024-04-13T13:10:42.243 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:10:42.270 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:10:42.270 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:10:42.279 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-04-13T13:10:42.305 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 564 kB/s | 90 kB 00:00 2024-04-13T13:10:42.308 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:42.308 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-04-13T13:10:42.374 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:42.386 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:42.386 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:42.480 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:42.480 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:42.763 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:10:42.764 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:10:42.991 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-13T13:10:43.223 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-13T13:10:43.278 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-13T13:10:43.316 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-13T13:10:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-13T13:10:43.451 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-13T13:10:43.517 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:10:43.560 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-13T13:10:43.593 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-13T13:10:43.608 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-13T13:10:43.631 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-13T13:10:43.755 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-13T13:10:43.799 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-13T13:10:43.982 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-13T13:10:44.015 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-13T13:10:44.101 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-13T13:10:44.328 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-13T13:10:44.381 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-13T13:10:44.414 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el9.noarch 13/13 2024-04-13T13:10:44.741 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-13T13:10:44.852 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-13T13:10:45.253 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el9.noarch 13/13 2024-04-13T13:10:45.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el9.noarch 1/13 2024-04-13T13:10:45.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-13T13:10:45.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-13T13:10:45.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-13T13:10:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-13T13:10:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:45.638 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-13T13:10:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-13T13:10:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:45.639 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:45.893 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-04-13T13:10:46.080 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-13T13:10:46.488 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:23 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.779 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:46.781 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.782 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.783 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.784 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:46.785 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.786 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:46.847 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 138 k 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-13T13:10:46.848 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-13T13:10:46.849 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:46.849 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:46.849 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:46.849 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-04-13T13:10:46.849 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:46.849 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 7.0 M 2024-04-13T13:10:46.850 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 33 M 2024-04-13T13:10:46.850 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:47.117 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-13T13:10:47.148 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540e 12/12 2024-04-13T13:10:47.691 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 925 kB/s | 248 kB 00:00 2024-04-13T13:10:47.749 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): ceph-mgr-cephadm-18.2.2-1187.gd540ebac.e 423 kB/s | 138 kB 00:00 2024-04-13T13:10:47.894 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-04-13T13:10:48.158 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 141 kB/s | 58 kB 00:00 2024-04-13T13:10:48.233 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-04-13T13:10:48.358 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 821 kB/s | 548 kB 00:00 2024-04-13T13:10:48.359 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:48.359 INFO:teuthology.orchestra.run.smithi122.stdout:Total 4.6 MB/s | 7.0 MB 00:01 2024-04-13T13:10:48.430 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:48.441 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:48.442 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:48.544 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:48.544 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:48.848 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:49.013 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-13T13:10:49.076 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-13T13:10:49.579 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-13T13:10:49.636 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540e 12/12 2024-04-13T13:10:49.636 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540e 1/12 2024-04-13T13:10:49.636 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-13T13:10:49.636 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-13T13:10:49.636 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-13T13:10:49.637 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-13T13:10:49.637 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-13T13:10:49.637 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-13T13:10:49.637 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-13T13:10:49.637 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-04-13T13:10:49.637 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-13T13:10:49.637 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-13T13:10:49.661 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-13T13:10:49.724 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-13T13:10:49.747 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 6/6 2024-04-13T13:10:50.142 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-13T13:10:50.142 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:50.142 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:10:50.142 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:50.143 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:50.517 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2024-04-13T13:10:50.568 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 6/6 2024-04-13T13:10:50.568 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/6 2024-04-13T13:10:50.569 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-13T13:10:50.569 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-13T13:10:50.569 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-13T13:10:50.569 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-13T13:10:50.889 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:50.890 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:51.144 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:20 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:51.148 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.447 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.448 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:51.449 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:51.450 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:51.451 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.452 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.453 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.454 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.455 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.456 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.456 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.456 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:51.456 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.456 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.456 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.456 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:51.519 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 49 k 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-13T13:10:51.520 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:51.521 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-04-13T13:10:51.522 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:51.522 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.2 M 2024-04-13T13:10:51.522 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2024-04-13T13:10:51.522 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:10:51.754 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:28 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:52.045 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:52.045 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:52.045 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:52.045 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.046 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:52.047 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.048 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.049 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.050 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.051 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.052 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.053 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.054 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:52.115 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:52.115 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:52.115 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:10:52.115 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:52.115 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:52.115 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 848 k 2024-04-13T13:10:52.116 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:10:52.116 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-13T13:10:52.116 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:52.116 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:52.116 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:52.116 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-04-13T13:10:52.116 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:52.117 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 928 k 2024-04-13T13:10:52.117 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.7 M 2024-04-13T13:10:52.117 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:52.215 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 69 kB/s | 19 kB 00:00 2024-04-13T13:10:52.241 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 85 kB/s | 26 kB 00:00 2024-04-13T13:10:52.266 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): ceph-mgr-rook-18.2.2-1187.gd540ebac.el9 151 kB/s | 49 kB 00:00 2024-04-13T13:10:52.591 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 453 kB/s | 159 kB 00:00 2024-04-13T13:10:52.650 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 512 kB/s | 222 kB 00:00 2024-04-13T13:10:52.700 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 494 kB/s | 54 kB 00:00 2024-04-13T13:10:52.743 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 220 kB/s | 80 kB 00:00 2024-04-13T13:10:52.810 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-18.2.2-1187.gd540ebac.el9.x86_ 1.9 MB/s | 848 kB 00:00 2024-04-13T13:10:52.810 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:52.811 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.3 MB/s | 928 kB 00:00 2024-04-13T13:10:52.822 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:52.844 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:52.845 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:52.921 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:52.921 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:52.934 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 417 kB/s | 279 kB 00:00 2024-04-13T13:10:52.959 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-04-13T13:10:52.984 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-04-13T13:10:53.121 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:53.240 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-13T13:10:53.284 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 579 kB/s | 203 kB 00:00 2024-04-13T13:10:53.318 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 2/2 2024-04-13T13:10:53.326 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-04-13T13:10:53.435 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-04-13T13:10:53.510 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-04-13T13:10:53.510 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:53.511 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.1 MB/s | 2.2 MB 00:01 2024-04-13T13:10:53.579 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:10:53.595 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:10:53.595 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:10:53.691 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:10:53.692 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:10:53.996 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 2/2 2024-04-13T13:10:53.997 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 1/2 2024-04-13T13:10:54.219 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:10:54.304 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-13T13:10:54.304 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:54.304 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:54.305 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-13T13:10:54.305 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:54.305 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:54.382 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-13T13:10:54.444 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-13T13:10:54.508 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-13T13:10:54.546 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-13T13:10:54.570 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-volume 2024-04-13T13:10:54.587 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-13T13:10:54.680 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-13T13:10:54.790 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-13T13:10:54.855 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-13T13:10:55.181 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:32 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:55.475 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:55.476 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.477 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.478 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.479 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.480 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.481 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.482 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.483 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.484 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.485 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:55.545 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:55.545 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:55.545 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 264 k 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 341 k 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.5 M 2024-04-13T13:10:55.546 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:55.575 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-13T13:10:55.619 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-13T13:10:55.653 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-13T13:10:55.701 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-13T13:10:55.733 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el9.noarch 13/13 2024-04-13T13:10:55.790 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 441 kB/s | 77 kB 00:00 2024-04-13T13:10:56.232 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-volume-18.2.2-1187.gd540ebac.el9.no 428 kB/s | 264 kB 00:00 2024-04-13T13:10:56.232 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:56.232 INFO:teuthology.orchestra.run.smithi122.stdout:Total 496 kB/s | 341 kB 00:00 2024-04-13T13:10:56.244 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:56.254 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:56.254 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:56.294 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:56.295 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:56.444 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el9.noarch 13/13 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el9.noarch 1/13 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-13T13:10:56.524 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-13T13:10:56.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-13T13:10:56.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-13T13:10:56.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-13T13:10:56.525 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-13T13:10:56.624 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-13T13:10:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:18.2.2-1187.gd540ebac.el9.noarch 2/2 2024-04-13T13:10:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:18.2.2-1187.gd540ebac.el9.noarch 2/2 2024-04-13T13:10:56.667 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:10:56.667 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-13T13:10:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:56.905 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-13T13:10:56.905 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:56.905 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:10:56.905 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:56.905 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:56.906 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:10:57.192 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2024-04-13T13:10:57.363 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:18.2.2-1187.gd540ebac.el9.noarch 1/2 2024-04-13T13:10:57.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-13T13:10:57.748 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:57.748 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:10:57.748 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:10:57.748 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-13T13:10:57.748 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:57.748 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:10:57.785 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:26 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:10:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:10:58.074 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:10:58.074 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:10:58.074 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.074 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:10:58.074 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.074 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:10:58.075 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.076 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.077 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.078 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.079 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.080 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.081 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.082 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:58.083 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.083 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.083 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.083 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-04-13T13:10:58.144 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:10:58.144 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:58.144 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 138 k 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:58.145 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:10:58.146 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:10:58.146 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-04-13T13:10:58.146 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:10:58.146 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 7.0 M 2024-04-13T13:10:58.146 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 33 M 2024-04-13T13:10:58.146 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:10:58.696 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-13T13:10:58.710 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:35 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:10:58.871 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-13T13:10:58.997 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:10:58.997 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:10:58.997 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:10:58.997 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.998 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:10:58.999 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:10:59.000 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:10:59.001 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:10:59.002 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.003 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.004 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.005 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.006 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:10:59.066 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:10:59.067 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:59.067 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-13T13:10:59.067 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:59.067 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:10:59.067 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 127 k 2024-04-13T13:10:59.067 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:59.067 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:10:59.068 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:10:59.068 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-13T13:10:59.068 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:10:59.068 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 127 k 2024-04-13T13:10:59.068 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 456 k 2024-04-13T13:10:59.068 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:10:59.105 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-04-13T13:10:59.130 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 223 kB/s | 58 kB 00:00 2024-04-13T13:10:59.155 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): ceph-mgr-cephadm-18.2.2-1187.gd540ebac.e 219 kB/s | 138 kB 00:00 2024-04-13T13:10:59.180 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 675 kB/s | 51 kB 00:00 2024-04-13T13:10:59.181 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:59.182 INFO:teuthology.orchestra.run.smithi083.stdout:Total 6.7 MB/s | 7.0 MB 00:01 2024-04-13T13:10:59.251 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:10:59.261 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:10:59.261 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:10:59.361 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:10:59.362 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:10:59.562 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-18.2.2-1187.gd540ebac.el9.x86_64 256 kB/s | 127 kB 00:00 2024-04-13T13:10:59.563 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:10:59.563 INFO:teuthology.orchestra.run.smithi122.stdout:Total 255 kB/s | 127 kB 00:00 2024-04-13T13:10:59.563 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:10:59.569 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:10:59.569 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:10:59.628 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:10:59.628 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:10:59.667 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:10:59.775 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:10:59.838 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-13T13:10:59.878 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:10:59.891 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-13T13:11:00.319 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:00.412 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-13T13:11:00.493 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-13T13:11:00.555 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-13T13:11:00.603 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 6/6 2024-04-13T13:11:00.653 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:00.653 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:00.653 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:11:00.653 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:00.653 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:00.653 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:00.882 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-04-13T13:11:01.377 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 6/6 2024-04-13T13:11:01.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 1/6 2024-04-13T13:11:01.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-13T13:11:01.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-13T13:11:01.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-13T13:11:01.378 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-13T13:11:01.499 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:38 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:01.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:01.707 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:01.790 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:01.790 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:01.790 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:01.790 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.791 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:01.792 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:01.793 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.794 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:01.795 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.796 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.797 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.798 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:01.799 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.800 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.800 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.800 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:01.800 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:01.859 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:01.860 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:11:01.860 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:01.933 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-04-13T13:11:01.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2024-04-13T13:11:02.533 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:39 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:02.563 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:31 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:02.823 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:02.823 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:02.823 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:02.823 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.823 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:02.823 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.823 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:02.823 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:02.824 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:02.825 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.826 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.827 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:02.828 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.830 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.831 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.832 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.852 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:02.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:02.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:02.855 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:02.856 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.857 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.858 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:02.859 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.860 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.861 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:02.893 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:02.894 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:02.894 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-13T13:11:02.894 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:02.894 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:11:02.894 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 31 k 2024-04-13T13:11:02.894 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:02.894 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:11:02.894 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:02.895 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-13T13:11:02.895 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:02.895 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 31 k 2024-04-13T13:11:02.895 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 139 k 2024-04-13T13:11:02.895 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:11:02.921 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:02.921 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 848 k 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-04-13T13:11:02.922 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:02.923 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 928 k 2024-04-13T13:11:02.923 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.7 M 2024-04-13T13:11:02.923 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:11:03.126 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-18.2.2-1187.gd540ebac.el9.x86_6 134 kB/s | 31 kB 00:00 2024-04-13T13:11:03.127 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:03.127 INFO:teuthology.orchestra.run.smithi122.stdout:Total 133 kB/s | 31 kB 00:00 2024-04-13T13:11:03.128 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:11:03.131 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:11:03.131 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:11:03.160 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:11:03.161 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:11:03.258 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:11:03.290 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): ceph-fuse-18.2.2-1187.gd540ebac.el9.x86_ 2.8 MB/s | 848 kB 00:00 2024-04-13T13:11:03.386 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:03.425 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 185 kB/s | 80 kB 00:00 2024-04-13T13:11:03.425 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:03.425 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.8 MB/s | 928 kB 00:00 2024-04-13T13:11:03.437 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:11:03.458 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:11:03.458 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:11:03.536 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:11:03.537 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:11:03.741 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:03.760 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:11:03.889 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-13T13:11:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 2/2 2024-04-13T13:11:04.026 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:04.026 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:04.026 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:11:04.026 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:04.026 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:04.027 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:04.196 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-04-13T13:11:04.661 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 2/2 2024-04-13T13:11:04.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 1/2 2024-04-13T13:11:04.799 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:41 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:05.005 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-13T13:11:05.005 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:05.006 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:11:05.006 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-13T13:11:05.006 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:05.006 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:05.087 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:05.088 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.089 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.090 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:05.091 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:05.092 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.093 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.094 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:05.095 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.096 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.097 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.097 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:05.097 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.097 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.097 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.097 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:05.097 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:05.156 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:05.157 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:11:05.157 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:05.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-04-13T13:11:05.277 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-volume 2024-04-13T13:11:05.860 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:42 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:05.891 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:34 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:06.147 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:06.147 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:06.148 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.149 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:06.150 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.151 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:06.152 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:06.153 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:06.154 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.155 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.157 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.157 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:06.177 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:06.177 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:06.177 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:06.177 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.177 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.178 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:06.179 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:06.180 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.181 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:06.182 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.183 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.185 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.186 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:06.215 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:06.216 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:11:06.216 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:06.245 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:06.246 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:06.246 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:18.2.2-1187.gd540ebac.el9 ceph-noarch 264 k 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 341 k 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.5 M 2024-04-13T13:11:06.247 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:11:06.294 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-04-13T13:11:06.496 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 544 kB/s | 77 kB 00:00 2024-04-13T13:11:06.713 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-volume-18.2.2-1187.gd540ebac.el9.no 735 kB/s | 264 kB 00:00 2024-04-13T13:11:06.713 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:06.713 INFO:teuthology.orchestra.run.smithi083.stdout:Total 731 kB/s | 341 kB 00:00 2024-04-13T13:11:06.725 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:11:06.735 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:11:06.735 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:11:06.778 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:11:06.779 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:11:06.891 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:43 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:06.930 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:11:07.111 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-13T13:11:07.142 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:18.2.2-1187.gd540ebac.el9.noarch 2/2 2024-04-13T13:11:07.158 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:18.2.2-1187.gd540ebac.el9.noarch 2/2 2024-04-13T13:11:07.158 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:11:07.158 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-13T13:11:07.158 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:07.178 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:07.178 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:07.178 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:07.178 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.178 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:07.178 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.178 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:07.178 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.179 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:07.180 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:07.181 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.182 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.183 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.184 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.185 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.186 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.187 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:07.188 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:07.248 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:07.249 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:11:07.249 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:07.327 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-04-13T13:11:07.892 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:18.2.2-1187.gd540ebac.el9.noarch 1/2 2024-04-13T13:11:07.925 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:44 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:08.212 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:08.212 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:08.213 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:08.214 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:08.215 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.216 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:08.217 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.218 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.219 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.220 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.221 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:08.222 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:08.239 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-13T13:11:08.239 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:08.239 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:11:08.239 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:18.2.2-1187.gd540ebac.el9.noarch 2024-04-13T13:11:08.239 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-13T13:11:08.239 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:08.239 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:08.280 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:08.281 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:11:08.281 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:08.359 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-04-13T13:11:08.594 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2024-04-13T13:11:08.960 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:45 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:09.204 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:38 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:09.251 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.252 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.253 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:09.254 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:09.255 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:09.256 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:09.257 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:09.258 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.259 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.260 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.261 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.261 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.261 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:09.261 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.261 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.261 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.261 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.261 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:09.322 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:09.323 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:11:09.323 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:09.397 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-04-13T13:11:09.494 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:09.494 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:09.494 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:09.494 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.494 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:09.494 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.494 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:09.494 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.495 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:09.496 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.497 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:09.498 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.499 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.500 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.501 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:09.502 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.503 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.503 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.503 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:09.564 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 127 k 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:09.565 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 127 k 2024-04-13T13:11:09.566 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 456 k 2024-04-13T13:11:09.566 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:11:09.771 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-18.2.2-1187.gd540ebac.el9.x86_64 616 kB/s | 127 kB 00:00 2024-04-13T13:11:09.772 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:09.772 INFO:teuthology.orchestra.run.smithi083.stdout:Total 612 kB/s | 127 kB 00:00 2024-04-13T13:11:09.772 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:11:09.777 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:11:09.777 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:11:09.838 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:11:09.838 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:11:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:11:09.995 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:46 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:10.102 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.282 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.283 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:10.284 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.285 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.287 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.288 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.289 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.290 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:10.291 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.291 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.291 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.291 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:10.291 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:10.351 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:10.352 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:11:10.352 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:10.433 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-04-13T13:11:10.551 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:10.838 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:10.838 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:10.838 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:11:10.838 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:10.838 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:10.838 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:11.030 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:48 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:11.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2024-04-13T13:11:11.319 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:11.320 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:11.321 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:11.322 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.323 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:11.324 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:11.325 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.326 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.327 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.328 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.329 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.329 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.329 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:11.329 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.329 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.329 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.329 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.391 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:11.392 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:11.392 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-04-13T13:11:11.392 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:11.392 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:11:11.392 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 87 k 2024-04-13T13:11:11.392 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:11.393 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:11:11.393 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:11.393 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-13T13:11:11.393 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:11.393 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 87 k 2024-04-13T13:11:11.393 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 227 k 2024-04-13T13:11:11.393 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:11:11.650 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:40 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:11.704 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-18.2.2-1187.gd540ebac.el9.x86_64.rpm 279 kB/s | 87 kB 00:00 2024-04-13T13:11:11.705 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:11.705 INFO:teuthology.orchestra.run.smithi122.stdout:Total 277 kB/s | 87 kB 00:00 2024-04-13T13:11:11.705 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:11:11.712 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:11:11.712 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:11:11.914 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:11:11.940 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:11.940 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:11.940 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:11.940 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.940 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.941 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.942 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.944 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:11.945 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.946 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.947 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.948 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.949 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:11.950 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:12.009 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:12.010 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:11:12.010 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:12.021 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:12.075 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2024-04-13T13:11:12.393 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:12.676 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:41 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:12.730 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:11:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:12.731 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:12.962 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:12.963 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:12.964 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.965 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:12.966 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:12.967 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.968 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:12.969 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.970 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.971 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.972 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:12.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-04-13T13:11:13.031 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:13.031 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 31 k 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-04-13T13:11:13.032 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:13.033 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 31 k 2024-04-13T13:11:13.033 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 139 k 2024-04-13T13:11:13.033 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:11:13.209 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-18.2.2-1187.gd540ebac.el9.x86_6 177 kB/s | 31 kB 00:00 2024-04-13T13:11:13.209 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:13.209 INFO:teuthology.orchestra.run.smithi083.stdout:Total 175 kB/s | 31 kB 00:00 2024-04-13T13:11:13.210 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:11:13.213 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:11:13.213 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:11:13.242 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:11:13.245 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:11:13.334 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:11:13.456 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:13.592 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:50 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:13.865 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.888 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.889 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:13.890 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.891 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.892 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.893 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.894 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.895 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:13.896 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.897 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:13.957 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:13.957 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:13.957 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 3.0 M 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.0 M 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 12 M 2024-04-13T13:11:13.958 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:11:14.134 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:14.135 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:14.135 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:11:14.135 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:14.135 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:14.135 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:14.325 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-18.2.2-1187.gd540ebac.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2024-04-13T13:11:14.325 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:14.326 INFO:teuthology.orchestra.run.smithi122.stdout:Total 8.1 MB/s | 3.0 MB 00:00 2024-04-13T13:11:14.326 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:11:14.334 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:11:14.335 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:11:14.354 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2024-04-13T13:11:14.410 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:11:14.411 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:11:14.826 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:11:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:14.901 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:11:14.901 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-13T13:11:14.902 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T13:11:14.902 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T13:11:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:14.957 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:43 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:15.247 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.248 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.249 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.250 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:15.251 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:15.252 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:15.253 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.254 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.255 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:15.256 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.257 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.257 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.257 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.257 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:15.257 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.257 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.257 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.257 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:15.258 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:15.317 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:15.318 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:11:15.318 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:15.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2024-04-13T13:11:15.986 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:44 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:16.012 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:16.013 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:16.013 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:11:16.013 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:16.013 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:16.013 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:16.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-04-13T13:11:16.275 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:16.275 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:16.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:16.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:16.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.275 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:16.275 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.276 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:16.277 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.278 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:16.279 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.280 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.281 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.282 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.283 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:16.284 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:16.344 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:16.345 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:11:16.345 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:16.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2024-04-13T13:11:16.867 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:53 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:17.014 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:46 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:17.155 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:17.155 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:17.155 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:17.156 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:17.157 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.158 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:17.159 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.160 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.161 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.162 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.163 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.164 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.164 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:17.164 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.164 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.164 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.164 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.223 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:17.223 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-04-13T13:11:17.223 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:17.223 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-13T13:11:17.224 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 172 k 2024-04-13T13:11:17.224 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:17.224 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-13T13:11:17.224 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-13T13:11:17.224 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-13T13:11:17.224 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:17.224 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 172 k 2024-04-13T13:11:17.225 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 494 k 2024-04-13T13:11:17.225 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-13T13:11:17.300 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:17.300 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:17.300 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:17.300 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.300 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:17.300 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.300 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:17.301 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:17.302 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.303 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.304 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:17.305 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.306 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.307 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.308 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.309 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:17.309 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:17.369 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:17.370 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:11:17.371 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:17.439 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2024-04-13T13:11:17.454 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-18.2.2-1187.gd540ebac.el9.x86_64.rpm 751 kB/s | 172 kB 00:00 2024-04-13T13:11:17.454 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:17.454 INFO:teuthology.orchestra.run.smithi122.stdout:Total 747 kB/s | 172 kB 00:00 2024-04-13T13:11:17.455 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-13T13:11:17.462 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-13T13:11:17.462 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-13T13:11:17.521 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-13T13:11:17.522 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-13T13:11:17.677 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-13T13:11:17.785 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:18.032 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:47 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:18.261 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:18.318 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:18.318 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.319 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:18.320 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.321 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.322 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.323 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:18.324 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.325 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.326 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.327 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.328 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:18.328 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.328 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.328 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.328 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:18.328 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:18.387 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:18.388 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:11:18.388 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:18.460 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2024-04-13T13:11:18.599 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:18.599 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:18.599 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-13T13:11:18.599 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:18.599 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:18.599 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:18.838 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-04-13T13:11:19.061 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:48 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.350 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:19.351 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.352 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.353 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:19.354 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:19.355 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.356 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.357 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.358 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.359 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.359 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:19.359 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.359 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.359 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.359 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.360 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:19.419 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:19.420 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:11:19.420 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:19.452 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:56 ago on Sat 13 Apr 2024 01:09:23 PM UTC. 2024-04-13T13:11:19.489 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2024-04-13T13:11:19.740 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-13T13:11:19.741 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-13T13:11:19.742 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.743 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-13T13:11:19.744 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-13T13:11:19.745 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:19.746 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.747 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:19.748 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.749 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.750 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:19.750 INFO:teuthology.orchestra.run.smithi122.stdout:Package rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:19.810 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-13T13:11:19.811 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-13T13:11:19.811 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-13T13:11:19.894 DEBUG:teuthology.parallel:result is None 2024-04-13T13:11:20.087 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:49 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.376 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.377 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:20.378 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:20.379 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:20.380 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.381 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.382 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.383 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.384 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:20.385 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:20.445 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:20.446 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:11:20.446 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:20.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2024-04-13T13:11:21.121 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:50 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:21.408 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:21.408 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:21.408 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:21.408 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.408 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.409 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.410 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.411 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:21.412 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:21.413 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.414 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:21.414 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.414 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:21.414 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.414 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.415 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.415 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:21.415 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.415 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.416 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.416 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.416 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:21.416 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.416 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.416 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.417 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.417 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.417 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.417 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:21.417 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.417 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.417 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.418 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.418 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:21.418 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.418 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.418 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.418 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.418 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.419 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.420 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:21.420 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.420 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.420 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.420 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:21.478 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-04-13T13:11:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:11:21.478 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 87 k 2024-04-13T13:11:21.478 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:11:21.479 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:21.479 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-04-13T13:11:21.479 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:21.479 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 87 k 2024-04-13T13:11:21.479 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 227 k 2024-04-13T13:11:21.479 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:11:21.695 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-18.2.2-1187.gd540ebac.el9.x86_64.rpm 403 kB/s | 87 kB 00:00 2024-04-13T13:11:21.695 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:21.695 INFO:teuthology.orchestra.run.smithi083.stdout:Total 400 kB/s | 87 kB 00:00 2024-04-13T13:11:21.695 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:11:21.702 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:11:21.702 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:11:21.764 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:11:21.764 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:11:21.909 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:11:22.011 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:22.419 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:22.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:22.709 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:22.709 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:11:22.709 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:22.709 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:22.709 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:22.944 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-04-13T13:11:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:52 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:23.845 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.846 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.847 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:23.848 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.849 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.850 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.851 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.852 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.854 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:23.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.854 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:23.913 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:23.914 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:23.914 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 3.0 M 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:23.915 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.0 M 2024-04-13T13:11:23.916 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 12 M 2024-04-13T13:11:23.916 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:11:24.932 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-18.2.2-1187.gd540ebac.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-04-13T13:11:24.933 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:24.933 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.9 MB/s | 3.0 MB 00:01 2024-04-13T13:11:24.934 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:11:24.944 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:11:24.945 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:11:25.024 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:11:25.025 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:11:25.443 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:11:25.508 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:25.533 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:25.533 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T13:11:25.534 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-13T13:11:25.534 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T13:11:25.534 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T13:11:25.534 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:26.640 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:26.641 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:26.641 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:11:26.641 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:26.641 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:26.641 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:26.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2024-04-13T13:11:27.513 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:56 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:27.806 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:27.806 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:27.806 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:27.806 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.806 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:27.806 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.806 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:27.807 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.808 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.809 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:27.810 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:27.811 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:27.812 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:27.813 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.814 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.815 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:27.876 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:27.876 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:18.2.2-1187.gd540ebac.el9 ceph 172 k 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-04-13T13:11:27.877 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:27.878 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 172 k 2024-04-13T13:11:27.878 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 494 k 2024-04-13T13:11:27.878 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-13T13:11:28.336 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-18.2.2-1187.gd540ebac.el9.x86_64.rpm 376 kB/s | 172 kB 00:00 2024-04-13T13:11:28.337 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-13T13:11:28.337 INFO:teuthology.orchestra.run.smithi083.stdout:Total 374 kB/s | 172 kB 00:00 2024-04-13T13:11:28.337 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-13T13:11:28.345 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-13T13:11:28.345 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-13T13:11:28.406 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-13T13:11:28.406 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-13T13:11:28.563 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-13T13:11:28.659 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:29.096 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:29.388 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:18.2.2-1187.gd540ebac.el9.x86_64 1/1 2024-04-13T13:11:29.389 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:29.389 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-04-13T13:11:29.389 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:18.2.2-1187.gd540ebac.el9.x86_64 2024-04-13T13:11:29.389 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:29.389 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:29.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-04-13T13:11:30.281 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:59 ago on Sat 13 Apr 2024 01:09:31 PM UTC. 2024-04-13T13:11:30.578 INFO:teuthology.orchestra.run.smithi083.stderr:Modular dependency problems: 2024-04-13T13:11:30.578 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-13T13:11:30.578 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 1: conflicting requests 2024-04-13T13:11:30.578 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.578 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 2: conflicting requests 2024-04-13T13:11:30.578 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 3: conflicting requests 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 4: conflicting requests 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 5: conflicting requests 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 6: conflicting requests 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 7: conflicting requests 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 8: conflicting requests 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 9: conflicting requests 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.579 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 10: conflicting requests 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 11: conflicting requests 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 12: conflicting requests 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 13: conflicting requests 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 14: conflicting requests 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 15: conflicting requests 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 16: conflicting requests 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.580 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 17: conflicting requests 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 18: conflicting requests 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 19: conflicting requests 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 20: conflicting requests 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 21: conflicting requests 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 22: conflicting requests 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 23: conflicting requests 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.581 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 24: conflicting requests 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 25: conflicting requests 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 26: conflicting requests 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 27: conflicting requests 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 28: conflicting requests 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 29: conflicting requests 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 30: conflicting requests 2024-04-13T13:11:30.582 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 31: conflicting requests 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 32: conflicting requests 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 33: conflicting requests 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 34: conflicting requests 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 35: conflicting requests 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 36: conflicting requests 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.583 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 37: conflicting requests 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 39: conflicting requests 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.584 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.585 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.586 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.587 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.587 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.587 INFO:teuthology.orchestra.run.smithi083.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T13:11:30.587 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.587 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.587 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.587 INFO:teuthology.orchestra.run.smithi083.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T13:11:30.587 INFO:teuthology.orchestra.run.smithi083.stdout:Package rbd-mirror-2:18.2.2-1187.gd540ebac.el9.x86_64 is already installed. 2024-04-13T13:11:30.647 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-13T13:11:30.649 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-13T13:11:30.649 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-13T13:11:30.716 DEBUG:teuthology.parallel:result is None 2024-04-13T13:11:30.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-13T13:11:30.958 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-13T13:11:30.979 INFO:teuthology.orchestra.run.smithi083.stdout:18.2.2-1187.gd540ebac.el9 2024-04-13T13:11:30.979 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1187.gd540ebac.el9 2024-04-13T13:11:30.980 INFO:teuthology.task.install:The correct ceph version 18.2.2-1187.gd540ebac is installed. 2024-04-13T13:11:30.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-13T13:11:31.130 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi122.stdout:18.2.2-1187.gd540ebac.el9 2024-04-13T13:11:31.154 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1187.gd540ebac.el9 2024-04-13T13:11:31.154 INFO:teuthology.task.install:The correct ceph version 18.2.2-1187.gd540ebac is installed. 2024-04-13T13:11:31.156 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-13T13:11:31.156 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:31.157 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-13T13:11:31.185 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:31.185 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-13T13:11:31.226 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-13T13:11:31.226 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:31.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-04-13T13:11:31.253 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-13T13:11:31.321 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:31.321 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-04-13T13:11:31.354 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-13T13:11:31.424 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-13T13:11:31.425 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:31.425 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-13T13:11:31.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-13T13:11:31.519 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:31.519 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-13T13:11:31.548 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-13T13:11:31.618 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-13T13:11:31.618 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:31.618 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-04-13T13:11:31.644 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-13T13:11:31.712 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:31.712 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-04-13T13:11:31.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-13T13:11:31.811 INFO:teuthology.run_tasks:Running task ceph... 2024-04-13T13:11:31.900 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-13T13:11:31.901 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2024-04-13T13:11:31.903 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-04-13T13:11:31.930 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-13T13:11:31.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-13T13:11:31.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-13T13:11:31.998 INFO:tasks.ceph:Creating extra log directories... 2024-04-13T13:11:31.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-13T13:11:32.038 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-13T13:11:32.080 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-04-13T13:11:32.080 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-13T13:11:32.081 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008', 'branch': 'reef', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7650008', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '4f70a9a90881107996c8b7a3e89ba8b32624713c'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9619, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'sleep_before_teardown': 0, 'subset': '90/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4f70a9a90881107996c8b7a3e89ba8b32624713c', 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG//M3DRroy6RXlxQxY3rAUPaE9ctyve7QrEx+XePnci2VHICPl1SRaHiQ5hVKBv5+qaWfUZ/wjpNTtD3aY7/4U=', 'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRJMbsaAkfeSQKoa4lOc+m7Cn7k5Jn7V7UwZkRynUaEfDiUib85HyYAqrj1IrJgBro/SyCaVsnB0Pgix2dZxZ8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-09_22:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-13T13:11:32.081 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-13T13:11:32.105 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-13T13:11:32.122 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-04-13T13:11:32.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-04-13T13:11:32.188 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:32.189 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-04-13T13:11:32.239 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-13T13:11:32.240 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-04-13 13:11:25.768243509 +0000 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-04-13 13:08:35.186541968 +0000 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-04-13 13:08:35.186541968 +0000 2024-04-13T13:11:32.294 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-04-13 13:08:35.186541968 +0000 2024-04-13T13:11:32.294 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-13T13:11:32.359 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-04-13T13:11:32.360 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-04-13T13:11:32.360 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000254389 s, 2.0 MB/s 2024-04-13T13:11:32.361 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-13T13:11:32.417 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-04-13T13:11:32.472 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-13T13:11:32.472 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T13:11:32.472 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-13T13:11:32.472 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T13:11:32.472 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T13:11:32.472 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-04-13 13:11:25.768243509 +0000 2024-04-13T13:11:32.472 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-04-13 13:08:35.536533654 +0000 2024-04-13T13:11:32.473 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-04-13 13:08:35.536533654 +0000 2024-04-13T13:11:32.473 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-04-13 13:08:35.536533654 +0000 2024-04-13T13:11:32.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-13T13:11:32.538 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-04-13T13:11:32.538 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-04-13T13:11:32.538 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.00025746 s, 2.0 MB/s 2024-04-13T13:11:32.539 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-13T13:11:32.596 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-04-13 13:11:25.769243485 +0000 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-04-13 13:08:35.864525862 +0000 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-04-13 13:08:35.864525862 +0000 2024-04-13T13:11:32.651 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-04-13 13:08:35.864525862 +0000 2024-04-13T13:11:32.652 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-13T13:11:32.717 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-04-13T13:11:32.717 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-04-13T13:11:32.717 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000192486 s, 2.7 MB/s 2024-04-13T13:11:32.719 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-13T13:11:32.775 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-04-13T13:11:32.829 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-13T13:11:32.830 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T13:11:32.830 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-13T13:11:32.830 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T13:11:32.830 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T13:11:32.830 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-04-13 13:11:25.769243485 +0000 2024-04-13T13:11:32.830 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-04-13 13:08:36.190518117 +0000 2024-04-13T13:11:32.830 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-04-13 13:08:36.190518117 +0000 2024-04-13T13:11:32.830 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-04-13 13:08:36.190518117 +0000 2024-04-13T13:11:32.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-13T13:11:32.895 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-04-13T13:11:32.895 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-04-13T13:11:32.895 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000191701 s, 2.7 MB/s 2024-04-13T13:11:32.896 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-13T13:11:32.953 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T13:11:32.953 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-13T13:11:32.953 INFO:tasks.ceph:Generating config... 2024-04-13T13:11:32.954 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-13T13:11:32.954 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-13T13:11:32.954 INFO:tasks.ceph:[client] log to stderr = False 2024-04-13T13:11:32.954 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-13T13:11:32.954 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-04-13T13:11:32.954 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:32.955 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:32.955 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:32.955 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:32.955 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:32.955 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:32.955 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:32.955 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:32.955 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-13T13:11:32.955 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-13T13:11:32.955 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-13T13:11:32.955 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-13T13:11:32.955 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-13T13:11:32.955 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-13T13:11:32.955 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-13T13:11:32.956 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-13T13:11:32.956 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-13T13:11:32.957 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-13T13:11:32.957 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-13T13:11:32.957 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-13T13:11:32.957 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-13T13:11:32.957 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-13T13:11:32.957 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-13T13:11:32.957 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-04-13T13:11:32.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-04-13T13:11:33.056 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/cluster1.keyring 2024-04-13T13:11:33.059 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-04-13T13:11:33.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-13T13:11:33.170 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.83')] 2024-04-13T13:11:33.171 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.83', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-13T13:11:33.172 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:33.172 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-13T13:11:33.226 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.83 --print /home/ubuntu/cephtest/cluster1.monmap 2024-04-13T13:11:33.304 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-04-13T13:11:33.304 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid ba1aae70-25a6-4b6d-86eb-89f270bb7c30 2024-04-13T13:11:33.304 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = pacific 2024-04-13T13:11:33.305 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2024-04-13T13:11:33.305 INFO:teuthology.orchestra.run.smithi083.stdout:fsid ba1aae70-25a6-4b6d-86eb-89f270bb7c30 2024-04-13T13:11:33.305 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2024-04-13T13:11:33.305352+0000 2024-04-13T13:11:33.305 INFO:teuthology.orchestra.run.smithi083.stdout:created 2024-04-13T13:11:33.305352+0000 2024-04-13T13:11:33.305 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 16 (pacific) 2024-04-13T13:11:33.305 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 1 2024-04-13T13:11:33.305 INFO:teuthology.orchestra.run.smithi083.stdout:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-04-13T13:11:33.305 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-04-13T13:11:33.306 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-13T13:11:33.361 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID ba1aae70-25a6-4b6d-86eb-89f270bb7c30... 2024-04-13T13:11:33.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-13T13:11:33.403 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-13T13:11:33.451 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-04-13T13:11:33.451 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-04-13T13:11:33.451 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-13T13:11:33.451 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-04-13T13:11:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-04-13T13:11:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = 172.21.15.83 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: mon client directed command retry = 5 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on pool no app = False 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject socket failures = 5000 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = ba1aae70-25a6-4b6d-86eb-89f270bb7c30 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-04-13T13:11:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = stupid 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-04-13T13:11:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-04-13T13:11:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-04-13T13:11:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.460 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-04-13T13:11:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: debug rbd = 20 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: debug rbd_mirror = 15 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: log to stderr = False 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: rbd default features = 125 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: rbd mirror image policy type = simple 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-04-13T13:11:33.462 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.0] 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.1] 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.2] 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.3] 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.464 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-04-13T13:11:33.465 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-04-13T13:11:33.465 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.465 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-04-13T13:11:33.465 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-04-13T13:11:33.465 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-04-13T13:11:33.466 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-04-13T13:11:33.467 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = 172.21.15.83 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on pool no app = False 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = ba1aae70-25a6-4b6d-86eb-89f270bb7c30 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-04-13T13:11:33.468 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-04-13T13:11:33.469 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-04-13T13:11:33.470 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-04-13T13:11:33.470 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.470 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-04-13T13:11:33.470 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-04-13T13:11:33.470 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.470 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-04-13T13:11:33.470 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.470 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = stupid 2024-04-13T13:11:33.471 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-04-13T13:11:33.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-04-13T13:11:33.473 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-04-13T13:11:33.474 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd = 20 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd_mirror = 15 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: log to stderr = False 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: rbd default features = 125 2024-04-13T13:11:33.475 INFO:teuthology.orchestra.run.smithi122.stdout: rbd mirror image policy type = simple 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:11:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:11:33.478 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-04-13T13:11:33.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-04-13T13:11:33.573 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-13T13:11:33.574 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:33.574 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-04-13T13:11:33.588 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:33.588 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-04-13T13:11:33.643 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T13:11:33.643 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:33.643 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-13T13:11:33.643 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-13T13:11:33.722 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:33.723 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-13T13:11:33.778 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T13:11:33.778 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:33.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-13T13:11:33.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-13T13:11:33.819 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:33.820 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-13T13:11:33.876 INFO:tasks.ceph:Setting up mon nodes... 2024-04-13T13:11:33.876 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-13T13:11:33.877 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-13T13:11:33.931 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-13T13:11:33.934 INFO:tasks.ceph:Setting up mds nodes... 2024-04-13T13:11:33.934 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-13T13:11:33.935 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-04-13T13:11:33.977 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-04-13T13:11:33.992 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-04-13T13:11:34.042 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-04-13T13:11:34.058 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-13T13:11:34.099 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-13T13:11:34.115 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-13T13:11:34.197 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-13T13:11:34.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-13T13:11:34.294 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-13T13:11:34.309 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-13T13:11:34.390 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-13T13:11:34.406 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-13T13:11:34.488 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-13T13:11:34.503 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-13T13:11:34.586 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-13T13:11:34.601 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-13T13:11:34.682 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-13T13:11:34.697 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-13T13:11:34.698 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-13T13:11:34.698 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-04-13T13:11:34.728 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T13:11:34.728 INFO:tasks.ceph:role: cluster1.osd.0 2024-04-13T13:11:34.728 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T13:11:34.728 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-13T13:11:34.805 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T13:11:34.805 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T13:11:34.805 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T13:11:34.805 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-13T13:11:34.805 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T13:11:34.805 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-04-13T13:11:34.805 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T13:11:34.806 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-13T13:11:34.806 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T13:11:34.806 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T13:11:35.300 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-04-13T13:11:35.302 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-04-13T13:11:35.302 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-04-13T13:11:35.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-04-13T13:11:35.471 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-04-13T13:11:35.542 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T13:11:35.542 INFO:tasks.ceph:role: cluster1.osd.1 2024-04-13T13:11:35.542 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T13:11:35.542 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-13T13:11:35.617 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T13:11:35.617 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T13:11:35.617 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T13:11:35.617 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-13T13:11:35.618 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T13:11:35.618 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-04-13T13:11:35.618 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T13:11:35.618 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-13T13:11:35.618 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T13:11:35.618 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T13:11:36.070 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-04-13T13:11:36.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-04-13T13:11:36.072 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-04-13T13:11:36.165 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-04-13T13:11:36.196 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-04-13T13:11:36.265 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T13:11:36.265 INFO:tasks.ceph:role: cluster1.osd.2 2024-04-13T13:11:36.265 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T13:11:36.265 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-13T13:11:36.339 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T13:11:36.340 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T13:11:36.836 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-04-13T13:11:36.860 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-04-13T13:11:36.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-04-13T13:11:36.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-04-13T13:11:37.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-13T13:11:37.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:37.125+0000 7f94d7b91640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-04-13T13:11:37.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:37.128+0000 7f94d7b91640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-04-13T13:11:37.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:37.128+0000 7f94d7b91640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-13T13:11:37.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:37.128+0000 7f94d7b91640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-13T13:11:37.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:37.128+0000 7f94d7b91640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-13T13:11:37.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:37.135+0000 7f94d7b91640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-04-13T13:11:39.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-04-13T13:11:40.009 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-13T13:11:40.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:40.098+0000 7f6b69a15640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-04-13T13:11:40.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:40.099+0000 7f6b69a15640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-04-13T13:11:40.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:40.099+0000 7f6b69a15640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-13T13:11:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:40.099+0000 7f6b69a15640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-13T13:11:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:40.099+0000 7f6b69a15640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-13T13:11:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:40.099+0000 7f6b69a15640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-04-13T13:11:42.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-04-13T13:11:42.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-13T13:11:43.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:43.079+0000 7f680c813640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-04-13T13:11:43.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:43.079+0000 7f680c813640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-04-13T13:11:43.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:43.079+0000 7f680c813640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-13T13:11:43.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:43.079+0000 7f680c813640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-13T13:11:43.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:43.079+0000 7f680c813640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-13T13:11:43.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-04-13T13:11:43.079+0000 7f680c813640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-04-13T13:11:45.926 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-04-13T13:11:45.956 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-13T13:11:45.957 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:45.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-04-13T13:11:46.025 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:46.025 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-04-13T13:11:46.093 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:46.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-04-13T13:11:46.161 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:46.161 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-04-13T13:11:46.230 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:46.230 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-04-13T13:11:46.286 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:46.286 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-04-13T13:11:46.308 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:46.308 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-04-13T13:11:46.366 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:46.366 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-04-13T13:11:46.424 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:46.424 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-04-13T13:11:46.481 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:46.482 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-04-13T13:11:46.540 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:46.540 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-04-13T13:11:46.599 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:46.599 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-04-13T13:11:46.656 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:11:46.656 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-04-13T13:11:46.714 INFO:tasks.ceph:Adding keys to all mons... 2024-04-13T13:11:46.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/cluster1.keyring 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCFhBpm91CSNxAAeehxfTs4rBiccf39Os7DXQ== 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCJhBpm7WaMBxAAAgWSOdmVuAEYqa9/a53Heg== 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCMhBpmsIjzBRAAGH/kbqBXlWBOTZe95SoOEg== 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCPhBpmCdHGBBAAJTbA8JnqL7uiWyLYZZFVew== 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-04-13T13:11:46.740 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCFhBpm3ulHOhAADLpNjRZO3LS2FGJRDdnAug== 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror] 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCGhBpmdp6TAhAAoqRyoEny34Wmj/g4ALs5mg== 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.0] 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCGhBpmbkUABhAAWYnqOaaEIn6eC95LxOZg8w== 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.1] 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCGhBpm8w3OCxAAk7KztZFSRG/XyLEPrho6Zw== 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.2] 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCGhBpmUcOYERAAYsZsMIww6qhoBUDjY7TZTA== 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.3] 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCGhBpmk2xZFxAA1iy++FjS7bmUeKtSQNCsDQ== 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.4] 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCGhBpmnEYmHRAAImBlJ1lyp23SyXN+TpI8Ew== 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.5] 2024-04-13T13:11:46.741 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCGhBpm1Kj+IhAAz86gfsLQVXRG9P5xDtLKzQ== 2024-04-13T13:11:46.742 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.6] 2024-04-13T13:11:46.742 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCGhBpmkYu/KBAAJb6Fb2QlFNLXF7z4GS+j3w== 2024-04-13T13:11:46.743 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-13T13:11:46.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T13:11:46.890 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T13:11:46.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T13:11:46.995 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.049 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.320 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.429 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:11:47.483 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-13T13:11:47.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-04-13T13:11:47.510 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-04-13T13:11:47.607 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-04-13T13:11:47.634 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-04-13T13:11:47.690 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-04-13T13:11:47.690 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-04-13T13:11:47.690 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-04-13T13:11:47.732 INFO:tasks.ceph.cluster1.mon.a:Started 2024-04-13T13:11:47.733 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-04-13T13:11:47.733 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-04-13T13:11:47.733 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-04-13T13:11:47.735 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-04-13T13:11:47.735 INFO:tasks.ceph:Setting crush tunables to default 2024-04-13T13:11:47.736 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster cluster1 osd crush tunables default 2024-04-13T13:11:47.910 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-04-13T13:11:47.921 INFO:tasks.ceph:check_enable_crimson: False 2024-04-13T13:11:47.921 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-04-13T13:11:47.922 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:47.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-04-13T13:11:47.948 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:47.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-04-13T13:11:48.016 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:11:48.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-04-13T13:11:48.083 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster cluster1 osd new 7d35fb3b-0f30-45b7-a191-252bb626391a 0 2024-04-13T13:11:48.304 INFO:teuthology.orchestra.run.smithi083.stdout:0 2024-04-13T13:11:48.316 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster cluster1 osd new c126ff47-39be-415d-a147-51c00ff4430a 1 2024-04-13T13:11:48.493 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:48.492+0000 7f4353211040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T13:11:48.496 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-04-13T13:11:48.508 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster cluster1 osd new df560f34-6ebd-441b-bc26-bffc0ef97557 2 2024-04-13T13:11:48.551 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:48.550+0000 7f4353211040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T13:11:48.700 INFO:teuthology.orchestra.run.smithi083.stdout:2 2024-04-13T13:11:48.711 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-04-13T13:11:48.712 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-04-13T13:11:48.755 INFO:tasks.ceph.cluster1.osd.0:Started 2024-04-13T13:11:48.755 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-04-13T13:11:48.756 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-04-13T13:11:48.757 INFO:tasks.ceph.cluster1.osd.1:Started 2024-04-13T13:11:48.758 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-04-13T13:11:48.758 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-04-13T13:11:48.760 INFO:tasks.ceph.cluster1.osd.2:Started 2024-04-13T13:11:48.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-13T13:11:48.933 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:11:48.933 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":5,"fsid":"ba1aae70-25a6-4b6d-86eb-89f270bb7c30","created":"2024-04-13T13:11:47.821698+0000","modified":"2024-04-13T13:11:48.700044+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T13:11:48.700044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d35fb3b-0f30-45b7-a191-252bb626391a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c126ff47-39be-415d-a147-51c00ff4430a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df560f34-6ebd-441b-bc26-bffc0ef97557","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:11:48.943 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-04-13T13:11:48.944 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-13T13:11:49.147 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.146+0000 7f4353211040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T13:11:49.198 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.197+0000 7f4353211040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T13:11:49.241 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.241+0000 7f4353211040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T13:11:49.246 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-13T13:11:49.302 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.302+0000 7f4353211040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T13:11:49.347 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.347+0000 7f4353211040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T13:11:49.360 INFO:tasks.ceph.cluster1.osd.1.smithi083.stderr:2024-04-13T13:11:49.359+0000 7fa835923640 -1 Falling back to public interface 2024-04-13T13:11:49.367 INFO:tasks.ceph.cluster1.osd.2.smithi083.stderr:2024-04-13T13:11:49.367+0000 7fa98b819640 -1 Falling back to public interface 2024-04-13T13:11:49.374 INFO:tasks.ceph.cluster1.osd.0.smithi083.stderr:2024-04-13T13:11:49.374+0000 7febf9224640 -1 Falling back to public interface 2024-04-13T13:11:49.407 INFO:teuthology.misc.health.smithi083.stdout: 2024-04-13T13:11:49.407 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":5,"fsid":"ba1aae70-25a6-4b6d-86eb-89f270bb7c30","created":"2024-04-13T13:11:47.821698+0000","modified":"2024-04-13T13:11:48.700044+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T13:11:48.700044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d35fb3b-0f30-45b7-a191-252bb626391a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c126ff47-39be-415d-a147-51c00ff4430a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df560f34-6ebd-441b-bc26-bffc0ef97557","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:11:49.417 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-13T13:11:49.471 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.470+0000 7f4353211040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T13:11:49.539 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.538+0000 7f4353211040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T13:11:49.584 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.583+0000 7f4353211040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T13:11:49.636 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.636+0000 7f4353211040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T13:11:49.684 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:49.683+0000 7f4353211040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T13:11:50.059 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.058+0000 7f4353211040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T13:11:50.192 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.191+0000 7f4353211040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T13:11:50.323 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.323+0000 7f4353211040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T13:11:50.369 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.368+0000 7f4353211040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T13:11:50.485 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.482+0000 7f4353211040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T13:11:50.577 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.577+0000 7f4353211040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T13:11:50.733 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.732+0000 7f4353211040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T13:11:50.776 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.776+0000 7f4353211040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T13:11:50.930 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:50.929+0000 7f4353211040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T13:11:51.756 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:51.756+0000 7f4353211040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T13:11:52.031 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:52.031+0000 7f4353211040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T13:11:52.084 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:52.084+0000 7f4353211040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T13:11:52.228 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:52.227+0000 7f4353211040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T13:11:52.852 INFO:tasks.ceph.cluster1.osd.1.smithi083.stderr:2024-04-13T13:11:52.852+0000 7fa835923640 -1 osd.1 0 log_to_monitors true 2024-04-13T13:11:52.879 INFO:tasks.ceph.cluster1.osd.2.smithi083.stderr:2024-04-13T13:11:52.879+0000 7fa98b819640 -1 osd.2 0 log_to_monitors true 2024-04-13T13:11:52.894 INFO:tasks.ceph.cluster1.osd.0.smithi083.stderr:2024-04-13T13:11:52.894+0000 7febf9224640 -1 osd.0 0 log_to_monitors true 2024-04-13T13:11:53.235 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:11:53.235+0000 7f42f3ab5640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-13T13:11:55.720 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-13T13:11:56.072 INFO:tasks.ceph.cluster1.osd.0.smithi083.stderr:2024-04-13T13:11:56.071+0000 7febf45f6640 -1 osd.0 0 waiting for initial osdmap 2024-04-13T13:11:56.080 INFO:tasks.ceph.cluster1.osd.1.smithi083.stderr:2024-04-13T13:11:56.080+0000 7fa830df6640 -1 osd.1 0 waiting for initial osdmap 2024-04-13T13:11:56.084 INFO:tasks.ceph.cluster1.osd.0.smithi083.stderr:2024-04-13T13:11:56.084+0000 7febecc83640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T13:11:56.101 INFO:tasks.ceph.cluster1.osd.1.smithi083.stderr:2024-04-13T13:11:56.101+0000 7fa829483640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T13:11:56.109 INFO:tasks.ceph.cluster1.osd.2.smithi083.stderr:2024-04-13T13:11:56.109+0000 7fa9867d4640 -1 osd.2 0 waiting for initial osdmap 2024-04-13T13:11:56.126 INFO:tasks.ceph.cluster1.osd.2.smithi083.stderr:2024-04-13T13:11:56.126+0000 7fa97ee07640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T13:11:56.126 INFO:teuthology.misc.health.smithi083.stdout: 2024-04-13T13:11:56.127 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":7,"fsid":"ba1aae70-25a6-4b6d-86eb-89f270bb7c30","created":"2024-04-13T13:11:47.821698+0000","modified":"2024-04-13T13:11:54.286013+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T13:11:48.700044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d35fb3b-0f30-45b7-a191-252bb626391a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c126ff47-39be-415d-a147-51c00ff4430a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df560f34-6ebd-441b-bc26-bffc0ef97557","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:11:56.137 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-13T13:12:02.440 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-13T13:12:02.723 INFO:teuthology.misc.health.smithi083.stdout: 2024-04-13T13:12:02.723 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"ba1aae70-25a6-4b6d-86eb-89f270bb7c30","created":"2024-04-13T13:11:47.821698+0000","modified":"2024-04-13T13:12:00.380767+0000","last_up_change":"2024-04-13T13:11:57.083606+0000","last_in_change":"2024-04-13T13:11:48.700044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T13:11:58.265097+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7d35fb3b-0f30-45b7-a191-252bb626391a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":33132},{"type":"v1","addr":"172.21.15.83:6817","nonce":33132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":33132},{"type":"v1","addr":"172.21.15.83:6819","nonce":33132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":33132},{"type":"v1","addr":"172.21.15.83:6823","nonce":33132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":33132},{"type":"v1","addr":"172.21.15.83:6821","nonce":33132}]},"public_addr":"172.21.15.83:6817/33132","cluster_addr":"172.21.15.83:6819/33132","heartbeat_back_addr":"172.21.15.83:6823/33132","heartbeat_front_addr":"172.21.15.83:6821/33132","state":["exists","up"]},{"osd":1,"uuid":"c126ff47-39be-415d-a147-51c00ff4430a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":33130},{"type":"v1","addr":"172.21.15.83:6801","nonce":33130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":33130},{"type":"v1","addr":"172.21.15.83:6803","nonce":33130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":33130},{"type":"v1","addr":"172.21.15.83:6807","nonce":33130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":33130},{"type":"v1","addr":"172.21.15.83:6805","nonce":33130}]},"public_addr":"172.21.15.83:6801/33130","cluster_addr":"172.21.15.83:6803/33130","heartbeat_back_addr":"172.21.15.83:6807/33130","heartbeat_front_addr":"172.21.15.83:6805/33130","state":["exists","up"]},{"osd":2,"uuid":"df560f34-6ebd-441b-bc26-bffc0ef97557","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":33131},{"type":"v1","addr":"172.21.15.83:6809","nonce":33131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":33131},{"type":"v1","addr":"172.21.15.83:6811","nonce":33131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":33131},{"type":"v1","addr":"172.21.15.83:6815","nonce":33131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":33131},{"type":"v1","addr":"172.21.15.83:6813","nonce":33131}]},"public_addr":"172.21.15.83:6809/33131","cluster_addr":"172.21.15.83:6811/33131","heartbeat_back_addr":"172.21.15.83:6815/33131","heartbeat_front_addr":"172.21.15.83:6813/33131","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.879258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.849495+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.903443+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:02.734 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-13T13:12:02.734 INFO:tasks.ceph:Creating RBD pool 2024-04-13T13:12:02.735 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-04-13T13:12:03.391 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'rbd' created 2024-04-13T13:12:03.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-13T13:12:04.411 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-04-13T13:12:04.411 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-04-13T13:12:04.412 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-04-13T13:12:04.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-04-13T13:12:04.412 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-13T13:12:04.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:04.751 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":32794},{"type":"v1","addr":"172.21.15.83:6825","nonce":32794}]},"active_addr":"172.21.15.83:6825/32794","active_change":"2024-04-13T13:11:52.229984+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":946196828}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2358313675}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3668967581}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3846896707}]}]} 2024-04-13T13:12:04.755 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-04-13T13:12:04.756 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-04-13T13:12:04.756 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-13T13:12:05.038 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:05.038 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"fsid":"ba1aae70-25a6-4b6d-86eb-89f270bb7c30","created":"2024-04-13T13:11:47.821698+0000","modified":"2024-04-13T13:12:04.389452+0000","last_up_change":"2024-04-13T13:11:57.083606+0000","last_in_change":"2024-04-13T13:11:48.700044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T13:11:58.265097+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-13T13:12:03.034420+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7d35fb3b-0f30-45b7-a191-252bb626391a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":33132},{"type":"v1","addr":"172.21.15.83:6817","nonce":33132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":33132},{"type":"v1","addr":"172.21.15.83:6819","nonce":33132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":33132},{"type":"v1","addr":"172.21.15.83:6823","nonce":33132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":33132},{"type":"v1","addr":"172.21.15.83:6821","nonce":33132}]},"public_addr":"172.21.15.83:6817/33132","cluster_addr":"172.21.15.83:6819/33132","heartbeat_back_addr":"172.21.15.83:6823/33132","heartbeat_front_addr":"172.21.15.83:6821/33132","state":["exists","up"]},{"osd":1,"uuid":"c126ff47-39be-415d-a147-51c00ff4430a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":33130},{"type":"v1","addr":"172.21.15.83:6801","nonce":33130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":33130},{"type":"v1","addr":"172.21.15.83:6803","nonce":33130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":33130},{"type":"v1","addr":"172.21.15.83:6807","nonce":33130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":33130},{"type":"v1","addr":"172.21.15.83:6805","nonce":33130}]},"public_addr":"172.21.15.83:6801/33130","cluster_addr":"172.21.15.83:6803/33130","heartbeat_back_addr":"172.21.15.83:6807/33130","heartbeat_front_addr":"172.21.15.83:6805/33130","state":["exists","up"]},{"osd":2,"uuid":"df560f34-6ebd-441b-bc26-bffc0ef97557","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":33131},{"type":"v1","addr":"172.21.15.83:6809","nonce":33131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":33131},{"type":"v1","addr":"172.21.15.83:6811","nonce":33131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":33131},{"type":"v1","addr":"172.21.15.83:6815","nonce":33131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":33131},{"type":"v1","addr":"172.21.15.83:6813","nonce":33131}]},"public_addr":"172.21.15.83:6809/33131","cluster_addr":"172.21.15.83:6811/33131","heartbeat_back_addr":"172.21.15.83:6815/33131","heartbeat_front_addr":"172.21.15.83:6813/33131","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.879258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.849495+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.903443+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:05.051 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-04-13T13:12:05.051 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-13T13:12:05.331 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:05.331 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"fsid":"ba1aae70-25a6-4b6d-86eb-89f270bb7c30","created":"2024-04-13T13:11:47.821698+0000","modified":"2024-04-13T13:12:04.389452+0000","last_up_change":"2024-04-13T13:11:57.083606+0000","last_in_change":"2024-04-13T13:11:48.700044+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T13:11:58.265097+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-13T13:12:03.034420+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7d35fb3b-0f30-45b7-a191-252bb626391a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":33132},{"type":"v1","addr":"172.21.15.83:6817","nonce":33132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":33132},{"type":"v1","addr":"172.21.15.83:6819","nonce":33132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":33132},{"type":"v1","addr":"172.21.15.83:6823","nonce":33132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":33132},{"type":"v1","addr":"172.21.15.83:6821","nonce":33132}]},"public_addr":"172.21.15.83:6817/33132","cluster_addr":"172.21.15.83:6819/33132","heartbeat_back_addr":"172.21.15.83:6823/33132","heartbeat_front_addr":"172.21.15.83:6821/33132","state":["exists","up"]},{"osd":1,"uuid":"c126ff47-39be-415d-a147-51c00ff4430a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":33130},{"type":"v1","addr":"172.21.15.83:6801","nonce":33130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":33130},{"type":"v1","addr":"172.21.15.83:6803","nonce":33130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":33130},{"type":"v1","addr":"172.21.15.83:6807","nonce":33130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":33130},{"type":"v1","addr":"172.21.15.83:6805","nonce":33130}]},"public_addr":"172.21.15.83:6801/33130","cluster_addr":"172.21.15.83:6803/33130","heartbeat_back_addr":"172.21.15.83:6807/33130","heartbeat_front_addr":"172.21.15.83:6805/33130","state":["exists","up"]},{"osd":2,"uuid":"df560f34-6ebd-441b-bc26-bffc0ef97557","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":33131},{"type":"v1","addr":"172.21.15.83:6809","nonce":33131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":33131},{"type":"v1","addr":"172.21.15.83:6811","nonce":33131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":33131},{"type":"v1","addr":"172.21.15.83:6815","nonce":33131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":33131},{"type":"v1","addr":"172.21.15.83:6813","nonce":33131}]},"public_addr":"172.21.15.83:6809/33131","cluster_addr":"172.21.15.83:6811/33131","heartbeat_back_addr":"172.21.15.83:6815/33131","heartbeat_front_addr":"172.21.15.83:6813/33131","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.879258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.849495+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:11:53.903443+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:05.344 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-04-13T13:12:05.344 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-04-13T13:12:05.345 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-04-13T13:12:05.483 INFO:teuthology.orchestra.run.smithi083.stdout:34359738373 2024-04-13T13:12:05.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-13T13:12:05.499 INFO:teuthology.orchestra.run.smithi083.stdout:34359738373 2024-04-13T13:12:05.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-13T13:12:05.500 INFO:teuthology.orchestra.run.smithi083.stdout:34359738373 2024-04-13T13:12:05.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-13T13:12:05.793 INFO:teuthology.orchestra.run.smithi083.stdout:34359738372 2024-04-13T13:12:05.806 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-04-13T13:12:05.842 INFO:teuthology.orchestra.run.smithi083.stdout:34359738372 2024-04-13T13:12:05.852 INFO:teuthology.orchestra.run.smithi083.stdout:34359738372 2024-04-13T13:12:05.855 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2024-04-13T13:12:05.864 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-04-13T13:12:06.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-13T13:12:06.857 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-13T13:12:06.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-13T13:12:07.126 INFO:teuthology.orchestra.run.smithi083.stdout:34359738373 2024-04-13T13:12:07.143 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2024-04-13T13:12:07.143 DEBUG:teuthology.parallel:result is None 2024-04-13T13:12:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:34359738373 2024-04-13T13:12:07.176 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2024-04-13T13:12:07.176 DEBUG:teuthology.parallel:result is None 2024-04-13T13:12:07.212 INFO:teuthology.orchestra.run.smithi083.stdout:34359738373 2024-04-13T13:12:07.224 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-04-13T13:12:07.224 DEBUG:teuthology.parallel:result is None 2024-04-13T13:12:07.225 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-13T13:12:07.225 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-13T13:12:07.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:07.542 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-04-13T13:12:07.553 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-13T13:12:06.235575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.845692"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405620+0000","last_change":"2024-04-13T13:12:04.403019+0000","last_active":"2024-04-13T13:12:04.405620+0000","last_peered":"2024-04-13T13:12:04.405620+0000","last_clean":"2024-04-13T13:12:04.405620+0000","last_became_active":"2024-04-13T13:12:04.402716+0000","last_became_peered":"2024-04-13T13:12:04.402716+0000","last_unstale":"2024-04-13T13:12:04.405620+0000","last_undegraded":"2024-04-13T13:12:04.405620+0000","last_fullsized":"2024-04-13T13:12:04.405620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T19:07:33.692841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405675+0000","last_change":"2024-04-13T13:12:04.403469+0000","last_active":"2024-04-13T13:12:04.405675+0000","last_peered":"2024-04-13T13:12:04.405675+0000","last_clean":"2024-04-13T13:12:04.405675+0000","last_became_active":"2024-04-13T13:12:04.403144+0000","last_became_peered":"2024-04-13T13:12:04.403144+0000","last_unstale":"2024-04-13T13:12:04.405675+0000","last_undegraded":"2024-04-13T13:12:04.405675+0000","last_fullsized":"2024-04-13T13:12:04.405675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T21:29:36.070599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405708+0000","last_change":"2024-04-13T13:12:04.404015+0000","last_active":"2024-04-13T13:12:04.405708+0000","last_peered":"2024-04-13T13:12:04.405708+0000","last_clean":"2024-04-13T13:12:04.405708+0000","last_became_active":"2024-04-13T13:12:04.403726+0000","last_became_peered":"2024-04-13T13:12:04.403726+0000","last_unstale":"2024-04-13T13:12:04.405708+0000","last_undegraded":"2024-04-13T13:12:04.405708+0000","last_fullsized":"2024-04-13T13:12:04.405708+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:08:59.456030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405554+0000","last_change":"2024-04-13T13:12:04.402975+0000","last_active":"2024-04-13T13:12:04.405554+0000","last_peered":"2024-04-13T13:12:04.405554+0000","last_clean":"2024-04-13T13:12:04.405554+0000","last_became_active":"2024-04-13T13:12:04.402681+0000","last_became_peered":"2024-04-13T13:12:04.402681+0000","last_unstale":"2024-04-13T13:12:04.405554+0000","last_undegraded":"2024-04-13T13:12:04.405554+0000","last_fullsized":"2024-04-13T13:12:04.405554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T16:11:46.562738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-13T13:12:05.394957+0000","last_change":"2024-04-13T13:12:04.403536+0000","last_active":"2024-04-13T13:12:05.394957+0000","last_peered":"2024-04-13T13:12:05.394957+0000","last_clean":"2024-04-13T13:12:05.394957+0000","last_became_active":"2024-04-13T13:12:04.403246+0000","last_became_peered":"2024-04-13T13:12:04.403246+0000","last_unstale":"2024-04-13T13:12:05.394957+0000","last_undegraded":"2024-04-13T13:12:05.394957+0000","last_fullsized":"2024-04-13T13:12:05.394957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T23:09:45.657639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405570+0000","last_change":"2024-04-13T13:12:04.402724+0000","last_active":"2024-04-13T13:12:04.405570+0000","last_peered":"2024-04-13T13:12:04.405570+0000","last_clean":"2024-04-13T13:12:04.405570+0000","last_became_active":"2024-04-13T13:12:04.402421+0000","last_became_peered":"2024-04-13T13:12:04.402421+0000","last_unstale":"2024-04-13T13:12:04.405570+0000","last_undegraded":"2024-04-13T13:12:04.405570+0000","last_fullsized":"2024-04-13T13:12:04.405570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T23:11:59.276870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405555+0000","last_change":"2024-04-13T13:12:04.402681+0000","last_active":"2024-04-13T13:12:04.405555+0000","last_peered":"2024-04-13T13:12:04.405555+0000","last_clean":"2024-04-13T13:12:04.405555+0000","last_became_active":"2024-04-13T13:12:04.402393+0000","last_became_peered":"2024-04-13T13:12:04.402393+0000","last_unstale":"2024-04-13T13:12:04.405555+0000","last_undegraded":"2024-04-13T13:12:04.405555+0000","last_fullsized":"2024-04-13T13:12:04.405555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T00:10:36.083204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.404619+0000","last_change":"2024-04-13T13:12:04.400068+0000","last_active":"2024-04-13T13:12:04.404619+0000","last_peered":"2024-04-13T13:12:04.404619+0000","last_clean":"2024-04-13T13:12:04.404619+0000","last_became_active":"2024-04-13T13:12:04.399803+0000","last_became_peered":"2024-04-13T13:12:04.399803+0000","last_unstale":"2024-04-13T13:12:04.404619+0000","last_undegraded":"2024-04-13T13:12:04.404619+0000","last_fullsized":"2024-04-13T13:12:04.404619+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T19:51:28.049706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T13:12:03.394249+0000","last_change":"2024-04-13T13:11:59.845731+0000","last_active":"2024-04-13T13:12:03.394249+0000","last_peered":"2024-04-13T13:12:03.394249+0000","last_clean":"2024-04-13T13:12:03.394249+0000","last_became_active":"2024-04-13T13:11:59.844771+0000","last_became_peered":"2024-04-13T13:11:59.844771+0000","last_unstale":"2024-04-13T13:12:03.394249+0000","last_undegraded":"2024-04-13T13:12:03.394249+0000","last_fullsized":"2024-04-13T13:12:03.394249+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:11:58.300609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:11:58.300609+0000","last_clean_scrub_stamp":"2024-04-13T13:11:58.300609+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:34:06.421284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T13:12:07.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-13T13:12:07.830 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:07.831 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-04-13T13:12:07.842 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-13T13:12:06.235575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.845692"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405620+0000","last_change":"2024-04-13T13:12:04.403019+0000","last_active":"2024-04-13T13:12:04.405620+0000","last_peered":"2024-04-13T13:12:04.405620+0000","last_clean":"2024-04-13T13:12:04.405620+0000","last_became_active":"2024-04-13T13:12:04.402716+0000","last_became_peered":"2024-04-13T13:12:04.402716+0000","last_unstale":"2024-04-13T13:12:04.405620+0000","last_undegraded":"2024-04-13T13:12:04.405620+0000","last_fullsized":"2024-04-13T13:12:04.405620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T19:07:33.692841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405675+0000","last_change":"2024-04-13T13:12:04.403469+0000","last_active":"2024-04-13T13:12:04.405675+0000","last_peered":"2024-04-13T13:12:04.405675+0000","last_clean":"2024-04-13T13:12:04.405675+0000","last_became_active":"2024-04-13T13:12:04.403144+0000","last_became_peered":"2024-04-13T13:12:04.403144+0000","last_unstale":"2024-04-13T13:12:04.405675+0000","last_undegraded":"2024-04-13T13:12:04.405675+0000","last_fullsized":"2024-04-13T13:12:04.405675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T21:29:36.070599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405708+0000","last_change":"2024-04-13T13:12:04.404015+0000","last_active":"2024-04-13T13:12:04.405708+0000","last_peered":"2024-04-13T13:12:04.405708+0000","last_clean":"2024-04-13T13:12:04.405708+0000","last_became_active":"2024-04-13T13:12:04.403726+0000","last_became_peered":"2024-04-13T13:12:04.403726+0000","last_unstale":"2024-04-13T13:12:04.405708+0000","last_undegraded":"2024-04-13T13:12:04.405708+0000","last_fullsized":"2024-04-13T13:12:04.405708+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:08:59.456030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405554+0000","last_change":"2024-04-13T13:12:04.402975+0000","last_active":"2024-04-13T13:12:04.405554+0000","last_peered":"2024-04-13T13:12:04.405554+0000","last_clean":"2024-04-13T13:12:04.405554+0000","last_became_active":"2024-04-13T13:12:04.402681+0000","last_became_peered":"2024-04-13T13:12:04.402681+0000","last_unstale":"2024-04-13T13:12:04.405554+0000","last_undegraded":"2024-04-13T13:12:04.405554+0000","last_fullsized":"2024-04-13T13:12:04.405554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T16:11:46.562738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-13T13:12:05.394957+0000","last_change":"2024-04-13T13:12:04.403536+0000","last_active":"2024-04-13T13:12:05.394957+0000","last_peered":"2024-04-13T13:12:05.394957+0000","last_clean":"2024-04-13T13:12:05.394957+0000","last_became_active":"2024-04-13T13:12:04.403246+0000","last_became_peered":"2024-04-13T13:12:04.403246+0000","last_unstale":"2024-04-13T13:12:05.394957+0000","last_undegraded":"2024-04-13T13:12:05.394957+0000","last_fullsized":"2024-04-13T13:12:05.394957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T23:09:45.657639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405570+0000","last_change":"2024-04-13T13:12:04.402724+0000","last_active":"2024-04-13T13:12:04.405570+0000","last_peered":"2024-04-13T13:12:04.405570+0000","last_clean":"2024-04-13T13:12:04.405570+0000","last_became_active":"2024-04-13T13:12:04.402421+0000","last_became_peered":"2024-04-13T13:12:04.402421+0000","last_unstale":"2024-04-13T13:12:04.405570+0000","last_undegraded":"2024-04-13T13:12:04.405570+0000","last_fullsized":"2024-04-13T13:12:04.405570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T23:11:59.276870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.405555+0000","last_change":"2024-04-13T13:12:04.402681+0000","last_active":"2024-04-13T13:12:04.405555+0000","last_peered":"2024-04-13T13:12:04.405555+0000","last_clean":"2024-04-13T13:12:04.405555+0000","last_became_active":"2024-04-13T13:12:04.402393+0000","last_became_peered":"2024-04-13T13:12:04.402393+0000","last_unstale":"2024-04-13T13:12:04.405555+0000","last_undegraded":"2024-04-13T13:12:04.405555+0000","last_fullsized":"2024-04-13T13:12:04.405555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T00:10:36.083204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:04.404619+0000","last_change":"2024-04-13T13:12:04.400068+0000","last_active":"2024-04-13T13:12:04.404619+0000","last_peered":"2024-04-13T13:12:04.404619+0000","last_clean":"2024-04-13T13:12:04.404619+0000","last_became_active":"2024-04-13T13:12:04.399803+0000","last_became_peered":"2024-04-13T13:12:04.399803+0000","last_unstale":"2024-04-13T13:12:04.404619+0000","last_undegraded":"2024-04-13T13:12:04.404619+0000","last_fullsized":"2024-04-13T13:12:04.404619+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:03.384992+0000","last_clean_scrub_stamp":"2024-04-13T13:12:03.384992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T19:51:28.049706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T13:12:03.394249+0000","last_change":"2024-04-13T13:11:59.845731+0000","last_active":"2024-04-13T13:12:03.394249+0000","last_peered":"2024-04-13T13:12:03.394249+0000","last_clean":"2024-04-13T13:12:03.394249+0000","last_became_active":"2024-04-13T13:11:59.844771+0000","last_became_peered":"2024-04-13T13:11:59.844771+0000","last_unstale":"2024-04-13T13:12:03.394249+0000","last_undegraded":"2024-04-13T13:12:03.394249+0000","last_fullsized":"2024-04-13T13:12:03.394249+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:11:58.300609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:11:58.300609+0000","last_clean_scrub_stamp":"2024-04-13T13:11:58.300609+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:34:06.421284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T13:12:07.843 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-13T13:12:07.843 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-04-13T13:12:07.843 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-04-13T13:12:07.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-04-13T13:12:08.157 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:08.157 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-13T13:12:08.169 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-04-13T13:12:08.170 INFO:teuthology.run_tasks:Running task ceph... 2024-04-13T13:12:08.179 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-04-13T13:12:08.179 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-13T13:12:08.179 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008', 'branch': 'reef', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7650008', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '4f70a9a90881107996c8b7a3e89ba8b32624713c'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9619, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'sleep_before_teardown': 0, 'subset': '90/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4f70a9a90881107996c8b7a3e89ba8b32624713c', 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG//M3DRroy6RXlxQxY3rAUPaE9ctyve7QrEx+XePnci2VHICPl1SRaHiQ5hVKBv5+qaWfUZ/wjpNTtD3aY7/4U=', 'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRJMbsaAkfeSQKoa4lOc+m7Cn7k5Jn7V7UwZkRynUaEfDiUib85HyYAqrj1IrJgBro/SyCaVsnB0Pgix2dZxZ8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-09_22:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-13T13:12:08.179 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-13T13:12:08.182 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-13T13:12:08.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-04-13T13:12:08.237 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-04-13T13:12:08.271 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:08.271 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-04-13T13:12:08.327 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-13T13:12:08.328 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-04-13T13:12:08.384 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-13T13:12:08.385 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T13:12:08.385 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-13T13:12:08.385 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T13:12:08.385 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T13:12:08.385 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-13 13:11:15.145368875 +0000 2024-04-13T13:12:08.385 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-13 13:08:36.080992764 +0000 2024-04-13T13:12:08.385 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-13 13:08:36.080992764 +0000 2024-04-13T13:12:08.385 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-13 13:08:36.080992764 +0000 2024-04-13T13:12:08.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-13T13:12:08.452 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-13T13:12:08.452 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-13T13:12:08.452 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000239453 s, 2.1 MB/s 2024-04-13T13:12:08.453 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-13T13:12:08.508 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-04-13T13:12:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-13T13:12:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T13:12:08.564 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-13T13:12:08.564 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T13:12:08.564 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T13:12:08.564 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-13 13:11:15.145368875 +0000 2024-04-13T13:12:08.565 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-13 13:08:36.420984389 +0000 2024-04-13T13:12:08.565 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-13 13:08:36.420984389 +0000 2024-04-13T13:12:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-13 13:08:36.420984389 +0000 2024-04-13T13:12:08.565 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-13T13:12:08.630 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-13T13:12:08.630 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-13T13:12:08.630 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000168214 s, 3.0 MB/s 2024-04-13T13:12:08.631 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-13T13:12:08.686 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-04-13T13:12:08.741 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-13T13:12:08.741 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T13:12:08.741 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-13T13:12:08.741 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T13:12:08.742 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T13:12:08.742 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-13 13:11:15.145368875 +0000 2024-04-13T13:12:08.742 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-13 13:08:36.759976039 +0000 2024-04-13T13:12:08.742 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-13 13:08:36.759976039 +0000 2024-04-13T13:12:08.742 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-13 13:08:36.759976039 +0000 2024-04-13T13:12:08.742 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-13T13:12:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-13T13:12:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-13T13:12:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000198556 s, 2.6 MB/s 2024-04-13T13:12:08.809 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-13T13:12:08.864 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-04-13T13:12:08.919 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-13T13:12:08.919 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T13:12:08.920 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-13T13:12:08.920 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T13:12:08.920 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T13:12:08.920 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-13 13:11:15.146368851 +0000 2024-04-13T13:12:08.920 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-13 13:08:37.088967936 +0000 2024-04-13T13:12:08.920 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-13 13:08:37.088967936 +0000 2024-04-13T13:12:08.920 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-13 13:08:37.088967936 +0000 2024-04-13T13:12:08.921 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-13T13:12:08.985 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-13T13:12:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-13T13:12:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000206733 s, 2.5 MB/s 2024-04-13T13:12:08.987 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-13T13:12:09.042 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T13:12:09.043 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-13T13:12:09.043 INFO:tasks.ceph:Generating config... 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client] log to stderr = False 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.044 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.045 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.045 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.045 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.045 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-13T13:12:09.045 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-13T13:12:09.045 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-13T13:12:09.045 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-13T13:12:09.045 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-13T13:12:09.045 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-13T13:12:09.045 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-13T13:12:09.045 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-13T13:12:09.045 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-13T13:12:09.045 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-13T13:12:09.046 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-13T13:12:09.047 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-04-13T13:12:09.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-04-13T13:12:09.124 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.keyring 2024-04-13T13:12:09.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-04-13T13:12:09.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-13T13:12:09.277 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.122')] 2024-04-13T13:12:09.278 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.122', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-13T13:12:09.279 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:09.279 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-13T13:12:09.333 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.122 --print /home/ubuntu/cephtest/cluster2.monmap 2024-04-13T13:12:09.411 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: generated fsid b4431297-1766-4dc7-94e8-9b7da8c7bf9c 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:setting min_mon_release = pacific 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:epoch 0 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:fsid b4431297-1766-4dc7-94e8-9b7da8c7bf9c 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:last_changed 2024-04-13T13:12:09.412589+0000 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:created 2024-04-13T13:12:09.412589+0000 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:min_mon_release 16 (pacific) 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:election_strategy: 1 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:0: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.a 2024-04-13T13:12:09.412 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-04-13T13:12:09.414 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-13T13:12:09.468 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID b4431297-1766-4dc7-94e8-9b7da8c7bf9c... 2024-04-13T13:12:09.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-13T13:12:09.472 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-13T13:12:09.526 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.527 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-04-13T13:12:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = 172.21.15.122 2024-04-13T13:12:09.529 INFO:teuthology.orchestra.run.smithi083.stdout: mon client directed command retry = 5 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on pool no app = False 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject socket failures = 5000 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = b4431297-1766-4dc7-94e8-9b7da8c7bf9c 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.530 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-04-13T13:12:09.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = stupid 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-04-13T13:12:09.532 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-04-13T13:12:09.533 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.534 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-04-13T13:12:09.535 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: debug rbd = 20 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: debug rbd_mirror = 15 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: log to stderr = False 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: rbd default features = 125 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: rbd mirror image policy type = simple 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.0] 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.1] 2024-04-13T13:12:09.536 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.537 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.537 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.2] 2024-04-13T13:12:09.537 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.537 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.537 INFO:teuthology.orchestra.run.smithi083.stdout:[client.mirror.3] 2024-04-13T13:12:09.537 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.537 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.561 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-04-13T13:12:09.561 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-04-13T13:12:09.561 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-13T13:12:09.561 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-04-13T13:12:09.561 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.561 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-04-13T13:12:09.561 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.561 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-04-13T13:12:09.562 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.563 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = 172.21.15.122 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on pool no app = False 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = b4431297-1766-4dc7-94e8-9b7da8c7bf9c 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-04-13T13:12:09.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-04-13T13:12:09.565 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = stupid 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-04-13T13:12:09.566 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.567 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-04-13T13:12:09.568 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-13T13:12:09.569 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd = 20 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd_mirror = 15 2024-04-13T13:12:09.570 INFO:teuthology.orchestra.run.smithi122.stdout: log to stderr = False 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: rbd default features = 125 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: rbd mirror image policy type = simple 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-13T13:12:09.571 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-13T13:12:09.575 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-04-13T13:12:09.575 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-04-13T13:12:09.670 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-13T13:12:09.671 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:09.671 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-04-13T13:12:09.685 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:09.686 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-04-13T13:12:09.741 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T13:12:09.741 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:12:09.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-13T13:12:09.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-13T13:12:09.787 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:12:09.787 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-13T13:12:09.843 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T13:12:09.843 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:09.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-13T13:12:09.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-13T13:12:09.882 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:09.882 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-13T13:12:09.938 INFO:tasks.ceph:Setting up mon nodes... 2024-04-13T13:12:09.938 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-13T13:12:09.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-13T13:12:09.996 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-13T13:12:09.998 INFO:tasks.ceph:Setting up mds nodes... 2024-04-13T13:12:09.998 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-13T13:12:09.999 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-04-13T13:12:10.041 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-04-13T13:12:10.056 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-04-13T13:12:10.097 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-04-13T13:12:10.113 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-13T13:12:10.194 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-13T13:12:10.209 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-13T13:12:10.292 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-13T13:12:10.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-13T13:12:10.389 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-13T13:12:10.404 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-13T13:12:10.486 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-13T13:12:10.501 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-13T13:12:10.582 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-13T13:12:10.598 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-13T13:12:10.680 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-13T13:12:10.695 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-13T13:12:10.779 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-13T13:12:10.794 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-13T13:12:10.794 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-13T13:12:10.794 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-04-13T13:12:10.861 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T13:12:10.861 INFO:tasks.ceph:role: cluster2.osd.0 2024-04-13T13:12:10.861 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T13:12:10.861 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-13T13:12:10.934 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T13:12:10.935 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T13:12:11.432 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-04-13T13:12:11.434 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-04-13T13:12:11.434 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-04-13T13:12:11.534 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-04-13T13:12:11.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-04-13T13:12:11.635 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T13:12:11.635 INFO:tasks.ceph:role: cluster2.osd.1 2024-04-13T13:12:11.635 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T13:12:11.635 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-13T13:12:11.709 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T13:12:11.709 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T13:12:11.709 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T13:12:11.710 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-13T13:12:11.710 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T13:12:11.710 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-04-13T13:12:11.710 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T13:12:11.710 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-13T13:12:11.710 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T13:12:11.710 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T13:12:12.174 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-04-13T13:12:12.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-04-13T13:12:12.175 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-04-13T13:12:12.269 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-04-13T13:12:12.302 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-04-13T13:12:12.370 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-13T13:12:12.370 INFO:tasks.ceph:role: cluster2.osd.2 2024-04-13T13:12:12.370 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T13:12:12.370 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-13T13:12:12.445 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-13T13:12:12.445 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-13T13:12:12.445 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-13T13:12:12.445 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-13T13:12:12.445 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-13T13:12:12.445 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-04-13T13:12:12.445 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-13T13:12:12.445 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-13T13:12:12.446 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-13T13:12:12.446 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-13T13:12:12.935 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-04-13T13:12:12.937 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-04-13T13:12:12.937 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-04-13T13:12:13.073 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-04-13T13:12:13.108 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-13T13:12:13.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:13.205+0000 7f37b0111640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-04-13T13:12:13.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:13.210+0000 7f37b0111640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-04-13T13:12:13.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:13.210+0000 7f37b0111640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-13T13:12:13.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:13.210+0000 7f37b0111640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-13T13:12:13.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:13.210+0000 7f37b0111640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-13T13:12:13.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:13.217+0000 7f37b0111640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-04-13T13:12:16.071 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-04-13T13:12:16.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-13T13:12:16.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:16.195+0000 7ff3dd60d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-04-13T13:12:16.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:16.196+0000 7ff3dd60d640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-04-13T13:12:16.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:16.196+0000 7ff3dd60d640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-13T13:12:16.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:16.196+0000 7ff3dd60d640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-13T13:12:16.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:16.196+0000 7ff3dd60d640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-13T13:12:16.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:16.196+0000 7ff3dd60d640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-04-13T13:12:19.044 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-04-13T13:12:19.074 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-13T13:12:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:19.163+0000 7f9a39d9a640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-04-13T13:12:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:19.164+0000 7f9a39d9a640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-04-13T13:12:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:19.164+0000 7f9a39d9a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-13T13:12:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:19.164+0000 7f9a39d9a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-13T13:12:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:19.164+0000 7f9a39d9a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-13T13:12:19.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-13T13:12:19.164+0000 7f9a39d9a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-04-13T13:12:22.019 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-04-13T13:12:22.052 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-13T13:12:22.052 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:12:22.052 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-04-13T13:12:22.084 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.084 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-04-13T13:12:22.123 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.123 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-04-13T13:12:22.195 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.195 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-04-13T13:12:22.264 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-13T13:12:22.265 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-04-13T13:12:22.283 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.283 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-04-13T13:12:22.321 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.322 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-04-13T13:12:22.378 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.378 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-04-13T13:12:22.435 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.435 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-04-13T13:12:22.491 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.491 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-04-13T13:12:22.548 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.548 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-04-13T13:12:22.604 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.604 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-04-13T13:12:22.660 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:22.660 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-04-13T13:12:22.715 INFO:tasks.ceph:Adding keys to all mons... 2024-04-13T13:12:22.716 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/cluster2.keyring 2024-04-13T13:12:22.781 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2024-04-13T13:12:22.781 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCphBpmnv5rOxAACSAcJK3jQ5UQJAh5ZSObuw== 2024-04-13T13:12:22.781 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-04-13T13:12:22.781 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCthBpmbWJjDBAAK+2IDKhKWpoVGJntBSpvzQ== 2024-04-13T13:12:22.781 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-04-13T13:12:22.781 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCwhBpmX+67CxAA1cbjkAKFSv9ar+dPaA3/cA== 2024-04-13T13:12:22.781 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCzhBpm7YPSCRAAtwyENWc7dv60rwFkOxWBog== 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpmqsOCAhAAKgufybftvfi5/IizU0cGWg== 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror] 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpmYy7hBRAAaOwmZB3dmsNX3kRwsa23Xg== 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpmMoysCxAA7kOeFH+MdPCsrr3A93hSQA== 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpmYCFyERAA2N9r8iZfkYquYJT/YCxV/g== 2024-04-13T13:12:22.782 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpmJClDFxAAbCzW8c5JK2Fbhgg+9AdCcg== 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpm5A8JHRAAzNGca7xEzNvzJG4XC8v42Q== 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.4] 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpmDYnLIhAA60U1ADFcOq5fvN7MnTHbTw== 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.5] 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpmydaWKBAA+z1enyjWu7SUw7l+AI/EQw== 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.6] 2024-04-13T13:12:22.783 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCqhBpmX2B7LhAAbj99nhLLyZuB1fTwwG+KqQ== 2024-04-13T13:12:22.784 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-13T13:12:22.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T13:12:22.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T13:12:22.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-13T13:12:23.034 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.089 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.143 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.415 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.469 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-13T13:12:23.523 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-13T13:12:23.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-04-13T13:12:23.551 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-04-13T13:12:23.649 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-04-13T13:12:23.676 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-04-13T13:12:23.731 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-04-13T13:12:23.731 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-04-13T13:12:23.731 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-04-13T13:12:23.773 INFO:tasks.ceph.cluster2.mon.a:Started 2024-04-13T13:12:23.774 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-04-13T13:12:23.774 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-04-13T13:12:23.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-04-13T13:12:23.777 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-04-13T13:12:23.777 INFO:tasks.ceph:Setting crush tunables to default 2024-04-13T13:12:23.777 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd crush tunables default 2024-04-13T13:12:23.952 INFO:teuthology.orchestra.run.smithi122.stderr:adjusted tunables profile to default 2024-04-13T13:12:23.963 INFO:tasks.ceph:check_enable_crimson: False 2024-04-13T13:12:23.963 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-04-13T13:12:23.963 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:23.963 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-04-13T13:12:23.990 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:23.990 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-04-13T13:12:24.057 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:12:24.057 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-04-13T13:12:24.123 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd new 33ec1da9-e2e5-4569-8159-846cd39d2ae9 0 2024-04-13T13:12:24.211 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.211+0000 7fa9de858040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T13:12:24.258 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.258+0000 7fa9de858040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T13:12:24.340 INFO:teuthology.orchestra.run.smithi122.stdout:0 2024-04-13T13:12:24.351 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd new 8212c95d-8244-4276-be0e-eebf1030b0fb 1 2024-04-13T13:12:24.528 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-04-13T13:12:24.539 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd new 780f03f0-18cc-4a76-ba38-dcb23e4ce630 2 2024-04-13T13:12:24.634 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.633+0000 7fa9de858040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T13:12:24.677 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.676+0000 7fa9de858040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T13:12:24.716 INFO:teuthology.orchestra.run.smithi122.stdout:2 2024-04-13T13:12:24.719 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.718+0000 7fa9de858040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T13:12:24.727 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-04-13T13:12:24.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-04-13T13:12:24.729 INFO:tasks.ceph.cluster2.osd.0:Started 2024-04-13T13:12:24.730 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-04-13T13:12:24.730 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-04-13T13:12:24.731 INFO:tasks.ceph.cluster2.osd.1:Started 2024-04-13T13:12:24.732 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-04-13T13:12:24.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-04-13T13:12:24.734 INFO:tasks.ceph.cluster2.osd.2:Started 2024-04-13T13:12:24.734 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-13T13:12:24.769 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.769+0000 7fa9de858040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T13:12:24.810 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.810+0000 7fa9de858040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T13:12:24.903 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:24.903 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":5,"fsid":"b4431297-1766-4dc7-94e8-9b7da8c7bf9c","created":"2024-04-13T13:12:23.861516+0000","modified":"2024-04-13T13:12:24.715619+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T13:12:24.715619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"33ec1da9-e2e5-4569-8159-846cd39d2ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8212c95d-8244-4276-be0e-eebf1030b0fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"780f03f0-18cc-4a76-ba38-dcb23e4ce630","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:24.914 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-04-13T13:12:24.914 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-13T13:12:24.921 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.921+0000 7fa9de858040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T13:12:24.974 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:24.974+0000 7fa9de858040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T13:12:25.015 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.015+0000 7fa9de858040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T13:12:25.064 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.063+0000 7fa9de858040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T13:12:25.106 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.105+0000 7fa9de858040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T13:12:25.216 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-13T13:12:25.328 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:12:25.328+0000 7fd28ce8b640 -1 Falling back to public interface 2024-04-13T13:12:25.332 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:12:25.332+0000 7f8118c0f640 -1 Falling back to public interface 2024-04-13T13:12:25.338 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:12:25.337+0000 7f592a60e640 -1 Falling back to public interface 2024-04-13T13:12:25.373 INFO:teuthology.misc.health.smithi122.stdout: 2024-04-13T13:12:25.374 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":5,"fsid":"b4431297-1766-4dc7-94e8-9b7da8c7bf9c","created":"2024-04-13T13:12:23.861516+0000","modified":"2024-04-13T13:12:24.715619+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T13:12:24.715619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"33ec1da9-e2e5-4569-8159-846cd39d2ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8212c95d-8244-4276-be0e-eebf1030b0fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"780f03f0-18cc-4a76-ba38-dcb23e4ce630","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:25.383 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-13T13:12:25.421 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.421+0000 7fa9de858040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T13:12:25.551 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.551+0000 7fa9de858040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T13:12:25.642 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.641+0000 7fa9de858040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T13:12:25.684 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.683+0000 7fa9de858040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T13:12:25.793 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.792+0000 7fa9de858040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T13:12:25.883 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:25.883+0000 7fa9de858040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T13:12:26.033 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:26.033+0000 7fa9de858040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T13:12:26.075 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:26.075+0000 7fa9de858040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T13:12:26.193 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:26.192+0000 7fa9de858040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T13:12:26.795 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:26.794+0000 7fa9de858040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T13:12:27.054 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:27.054+0000 7fa9de858040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T13:12:27.103 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:27.103+0000 7fa9de858040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T13:12:27.216 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:27.215+0000 7fa9de858040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T13:12:28.424 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:12:28.424+0000 7fa97fbe2640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-13T13:12:28.752 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:12:28.751+0000 7fd28ce8b640 -1 osd.0 0 log_to_monitors true 2024-04-13T13:12:28.822 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:12:28.822+0000 7f8118c0f640 -1 osd.1 0 log_to_monitors true 2024-04-13T13:12:28.827 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:12:28.827+0000 7f592a60e640 -1 osd.2 0 log_to_monitors true 2024-04-13T13:12:31.687 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-13T13:12:32.030 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:12:32.029+0000 7f8113bf6640 -1 osd.1 0 waiting for initial osdmap 2024-04-13T13:12:32.047 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:12:32.047+0000 7f810c229640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T13:12:32.082 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:12:32.082+0000 7f59253f6640 -1 osd.2 0 waiting for initial osdmap 2024-04-13T13:12:32.088 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:12:32.088+0000 7fd289bf9640 -1 osd.0 0 waiting for initial osdmap 2024-04-13T13:12:32.103 INFO:teuthology.misc.health.smithi122.stdout: 2024-04-13T13:12:32.103 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":7,"fsid":"b4431297-1766-4dc7-94e8-9b7da8c7bf9c","created":"2024-04-13T13:12:23.861516+0000","modified":"2024-04-13T13:12:30.223883+0000","last_up_change":"0.000000","last_in_change":"2024-04-13T13:12:24.715619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"33ec1da9-e2e5-4569-8159-846cd39d2ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8212c95d-8244-4276-be0e-eebf1030b0fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"780f03f0-18cc-4a76-ba38-dcb23e4ce630","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:32.103 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:12:32.103+0000 7f591da83640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T13:12:32.111 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:12:32.111+0000 7fd280a29640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T13:12:32.115 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-13T13:12:38.418 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-13T13:12:38.705 INFO:teuthology.misc.health.smithi122.stdout: 2024-04-13T13:12:38.705 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4431297-1766-4dc7-94e8-9b7da8c7bf9c","created":"2024-04-13T13:12:23.861516+0000","modified":"2024-04-13T13:12:36.039298+0000","last_up_change":"2024-04-13T13:12:33.031793+0000","last_in_change":"2024-04-13T13:12:24.715619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T13:12:33.252387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33ec1da9-e2e5-4569-8159-846cd39d2ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":33893},{"type":"v1","addr":"172.21.15.122:6801","nonce":33893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":33893},{"type":"v1","addr":"172.21.15.122:6803","nonce":33893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":33893},{"type":"v1","addr":"172.21.15.122:6807","nonce":33893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":33893},{"type":"v1","addr":"172.21.15.122:6805","nonce":33893}]},"public_addr":"172.21.15.122:6801/33893","cluster_addr":"172.21.15.122:6803/33893","heartbeat_back_addr":"172.21.15.122:6807/33893","heartbeat_front_addr":"172.21.15.122:6805/33893","state":["exists","up"]},{"osd":1,"uuid":"8212c95d-8244-4276-be0e-eebf1030b0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":33894},{"type":"v1","addr":"172.21.15.122:6809","nonce":33894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":33894},{"type":"v1","addr":"172.21.15.122:6811","nonce":33894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":33894},{"type":"v1","addr":"172.21.15.122:6815","nonce":33894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":33894},{"type":"v1","addr":"172.21.15.122:6813","nonce":33894}]},"public_addr":"172.21.15.122:6809/33894","cluster_addr":"172.21.15.122:6811/33894","heartbeat_back_addr":"172.21.15.122:6815/33894","heartbeat_front_addr":"172.21.15.122:6813/33894","state":["exists","up"]},{"osd":2,"uuid":"780f03f0-18cc-4a76-ba38-dcb23e4ce630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":33895},{"type":"v1","addr":"172.21.15.122:6817","nonce":33895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":33895},{"type":"v1","addr":"172.21.15.122:6819","nonce":33895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":33895},{"type":"v1","addr":"172.21.15.122:6823","nonce":33895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":33895},{"type":"v1","addr":"172.21.15.122:6821","nonce":33895}]},"public_addr":"172.21.15.122:6817/33895","cluster_addr":"172.21.15.122:6819/33895","heartbeat_back_addr":"172.21.15.122:6823/33895","heartbeat_front_addr":"172.21.15.122:6821/33895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.792988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.802320+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.792235+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:38.716 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-13T13:12:38.716 INFO:tasks.ceph:Creating RBD pool 2024-04-13T13:12:38.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-04-13T13:12:39.046 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'rbd' created 2024-04-13T13:12:39.061 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-13T13:12:40.056 INFO:teuthology.orchestra.run.smithi122.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-13T13:12:40.074 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-04-13T13:12:40.075 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-04-13T13:12:40.075 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-04-13T13:12:40.075 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-04-13T13:12:40.076 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-13T13:12:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:40.420 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":38524},{"type":"v1","addr":"172.21.15.83:6827","nonce":38524}]},"active_addr":"172.21.15.83:6827/38524","active_change":"2024-04-13T13:12:27.218037+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1646030337}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1242316033}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3283635330}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2798808895}]}]} 2024-04-13T13:12:40.422 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-04-13T13:12:40.422 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-04-13T13:12:40.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-13T13:12:40.702 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:40.702 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":13,"fsid":"b4431297-1766-4dc7-94e8-9b7da8c7bf9c","created":"2024-04-13T13:12:23.861516+0000","modified":"2024-04-13T13:12:40.048827+0000","last_up_change":"2024-04-13T13:12:33.031793+0000","last_in_change":"2024-04-13T13:12:24.715619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T13:12:33.252387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-13T13:12:39.017207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33ec1da9-e2e5-4569-8159-846cd39d2ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":33893},{"type":"v1","addr":"172.21.15.122:6801","nonce":33893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":33893},{"type":"v1","addr":"172.21.15.122:6803","nonce":33893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":33893},{"type":"v1","addr":"172.21.15.122:6807","nonce":33893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":33893},{"type":"v1","addr":"172.21.15.122:6805","nonce":33893}]},"public_addr":"172.21.15.122:6801/33893","cluster_addr":"172.21.15.122:6803/33893","heartbeat_back_addr":"172.21.15.122:6807/33893","heartbeat_front_addr":"172.21.15.122:6805/33893","state":["exists","up"]},{"osd":1,"uuid":"8212c95d-8244-4276-be0e-eebf1030b0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":33894},{"type":"v1","addr":"172.21.15.122:6809","nonce":33894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":33894},{"type":"v1","addr":"172.21.15.122:6811","nonce":33894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":33894},{"type":"v1","addr":"172.21.15.122:6815","nonce":33894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":33894},{"type":"v1","addr":"172.21.15.122:6813","nonce":33894}]},"public_addr":"172.21.15.122:6809/33894","cluster_addr":"172.21.15.122:6811/33894","heartbeat_back_addr":"172.21.15.122:6815/33894","heartbeat_front_addr":"172.21.15.122:6813/33894","state":["exists","up"]},{"osd":2,"uuid":"780f03f0-18cc-4a76-ba38-dcb23e4ce630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":33895},{"type":"v1","addr":"172.21.15.122:6817","nonce":33895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":33895},{"type":"v1","addr":"172.21.15.122:6819","nonce":33895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":33895},{"type":"v1","addr":"172.21.15.122:6823","nonce":33895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":33895},{"type":"v1","addr":"172.21.15.122:6821","nonce":33895}]},"public_addr":"172.21.15.122:6817/33895","cluster_addr":"172.21.15.122:6819/33895","heartbeat_back_addr":"172.21.15.122:6823/33895","heartbeat_front_addr":"172.21.15.122:6821/33895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.792988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.802320+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.792235+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:40.715 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-04-13T13:12:40.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-13T13:12:40.996 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:40.996 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":13,"fsid":"b4431297-1766-4dc7-94e8-9b7da8c7bf9c","created":"2024-04-13T13:12:23.861516+0000","modified":"2024-04-13T13:12:40.048827+0000","last_up_change":"2024-04-13T13:12:33.031793+0000","last_in_change":"2024-04-13T13:12:24.715619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T13:12:33.252387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-13T13:12:39.017207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33ec1da9-e2e5-4569-8159-846cd39d2ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":33893},{"type":"v1","addr":"172.21.15.122:6801","nonce":33893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":33893},{"type":"v1","addr":"172.21.15.122:6803","nonce":33893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":33893},{"type":"v1","addr":"172.21.15.122:6807","nonce":33893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":33893},{"type":"v1","addr":"172.21.15.122:6805","nonce":33893}]},"public_addr":"172.21.15.122:6801/33893","cluster_addr":"172.21.15.122:6803/33893","heartbeat_back_addr":"172.21.15.122:6807/33893","heartbeat_front_addr":"172.21.15.122:6805/33893","state":["exists","up"]},{"osd":1,"uuid":"8212c95d-8244-4276-be0e-eebf1030b0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":33894},{"type":"v1","addr":"172.21.15.122:6809","nonce":33894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":33894},{"type":"v1","addr":"172.21.15.122:6811","nonce":33894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":33894},{"type":"v1","addr":"172.21.15.122:6815","nonce":33894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":33894},{"type":"v1","addr":"172.21.15.122:6813","nonce":33894}]},"public_addr":"172.21.15.122:6809/33894","cluster_addr":"172.21.15.122:6811/33894","heartbeat_back_addr":"172.21.15.122:6815/33894","heartbeat_front_addr":"172.21.15.122:6813/33894","state":["exists","up"]},{"osd":2,"uuid":"780f03f0-18cc-4a76-ba38-dcb23e4ce630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":33895},{"type":"v1","addr":"172.21.15.122:6817","nonce":33895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":33895},{"type":"v1","addr":"172.21.15.122:6819","nonce":33895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":33895},{"type":"v1","addr":"172.21.15.122:6823","nonce":33895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":33895},{"type":"v1","addr":"172.21.15.122:6821","nonce":33895}]},"public_addr":"172.21.15.122:6817/33895","cluster_addr":"172.21.15.122:6819/33895","heartbeat_back_addr":"172.21.15.122:6823/33895","heartbeat_front_addr":"172.21.15.122:6821/33895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.792988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.802320+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T13:12:29.792235+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T13:12:41.008 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-04-13T13:12:41.009 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-04-13T13:12:41.009 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-04-13T13:12:41.149 INFO:teuthology.orchestra.run.smithi122.stdout:34359738373 2024-04-13T13:12:41.149 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-13T13:12:41.152 INFO:teuthology.orchestra.run.smithi122.stdout:34359738374 2024-04-13T13:12:41.152 INFO:teuthology.orchestra.run.smithi122.stdout:34359738372 2024-04-13T13:12:41.153 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-13T13:12:41.154 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-13T13:12:41.442 INFO:teuthology.orchestra.run.smithi122.stdout:34359738372 2024-04-13T13:12:41.458 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-04-13T13:12:41.486 INFO:teuthology.orchestra.run.smithi122.stdout:34359738371 2024-04-13T13:12:41.491 INFO:teuthology.orchestra.run.smithi122.stdout:34359738373 2024-04-13T13:12:41.499 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.2 2024-04-13T13:12:41.504 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-04-13T13:12:42.459 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-13T13:12:42.499 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-13T13:12:42.504 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-13T13:12:42.782 INFO:teuthology.orchestra.run.smithi122.stdout:34359738373 2024-04-13T13:12:42.795 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.0 2024-04-13T13:12:42.795 DEBUG:teuthology.parallel:result is None 2024-04-13T13:12:42.809 INFO:teuthology.orchestra.run.smithi122.stdout:34359738372 2024-04-13T13:12:42.815 INFO:teuthology.orchestra.run.smithi122.stdout:34359738374 2024-04-13T13:12:42.822 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.2 2024-04-13T13:12:42.822 DEBUG:teuthology.parallel:result is None 2024-04-13T13:12:42.828 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-04-13T13:12:42.828 DEBUG:teuthology.parallel:result is None 2024-04-13T13:12:42.828 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-13T13:12:42.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-13T13:12:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:43.148 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-04-13T13:12:43.160 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-13T13:12:41.223571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.177668"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.068929+0000","last_change":"2024-04-13T13:12:40.063803+0000","last_active":"2024-04-13T13:12:40.068929+0000","last_peered":"2024-04-13T13:12:40.068929+0000","last_clean":"2024-04-13T13:12:40.068929+0000","last_became_active":"2024-04-13T13:12:40.063533+0000","last_became_peered":"2024-04-13T13:12:40.063533+0000","last_unstale":"2024-04-13T13:12:40.068929+0000","last_undegraded":"2024-04-13T13:12:40.068929+0000","last_fullsized":"2024-04-13T13:12:40.068929+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T20:58:21.866563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.068914+0000","last_change":"2024-04-13T13:12:40.063540+0000","last_active":"2024-04-13T13:12:40.068914+0000","last_peered":"2024-04-13T13:12:40.068914+0000","last_clean":"2024-04-13T13:12:40.068914+0000","last_became_active":"2024-04-13T13:12:40.063264+0000","last_became_peered":"2024-04-13T13:12:40.063264+0000","last_unstale":"2024-04-13T13:12:40.068914+0000","last_undegraded":"2024-04-13T13:12:40.068914+0000","last_fullsized":"2024-04-13T13:12:40.068914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T20:41:39.391010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.068921+0000","last_change":"2024-04-13T13:12:40.063758+0000","last_active":"2024-04-13T13:12:40.068921+0000","last_peered":"2024-04-13T13:12:40.068921+0000","last_clean":"2024-04-13T13:12:40.068921+0000","last_became_active":"2024-04-13T13:12:40.063310+0000","last_became_peered":"2024-04-13T13:12:40.063310+0000","last_unstale":"2024-04-13T13:12:40.068921+0000","last_undegraded":"2024-04-13T13:12:40.068921+0000","last_fullsized":"2024-04-13T13:12:40.068921+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:58:37.147448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.068993+0000","last_change":"2024-04-13T13:12:40.064229+0000","last_active":"2024-04-13T13:12:40.068993+0000","last_peered":"2024-04-13T13:12:40.068993+0000","last_clean":"2024-04-13T13:12:40.068993+0000","last_became_active":"2024-04-13T13:12:40.063910+0000","last_became_peered":"2024-04-13T13:12:40.063910+0000","last_unstale":"2024-04-13T13:12:40.068993+0000","last_undegraded":"2024-04-13T13:12:40.068993+0000","last_fullsized":"2024-04-13T13:12:40.068993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T00:43:25.158364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-13T13:12:41.064368+0000","last_change":"2024-04-13T13:12:40.064583+0000","last_active":"2024-04-13T13:12:41.064368+0000","last_peered":"2024-04-13T13:12:41.064368+0000","last_clean":"2024-04-13T13:12:41.064368+0000","last_became_active":"2024-04-13T13:12:40.064295+0000","last_became_peered":"2024-04-13T13:12:40.064295+0000","last_unstale":"2024-04-13T13:12:41.064368+0000","last_undegraded":"2024-04-13T13:12:41.064368+0000","last_fullsized":"2024-04-13T13:12:41.064368+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T00:46:27.385820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.065070+0000","last_change":"2024-04-13T13:12:40.063508+0000","last_active":"2024-04-13T13:12:40.065070+0000","last_peered":"2024-04-13T13:12:40.065070+0000","last_clean":"2024-04-13T13:12:40.065070+0000","last_became_active":"2024-04-13T13:12:40.063218+0000","last_became_peered":"2024-04-13T13:12:40.063218+0000","last_unstale":"2024-04-13T13:12:40.065070+0000","last_undegraded":"2024-04-13T13:12:40.065070+0000","last_fullsized":"2024-04-13T13:12:40.065070+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:41.174354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.065063+0000","last_change":"2024-04-13T13:12:40.063540+0000","last_active":"2024-04-13T13:12:40.065063+0000","last_peered":"2024-04-13T13:12:40.065063+0000","last_clean":"2024-04-13T13:12:40.065063+0000","last_became_active":"2024-04-13T13:12:40.063266+0000","last_became_peered":"2024-04-13T13:12:40.063266+0000","last_unstale":"2024-04-13T13:12:40.065063+0000","last_undegraded":"2024-04-13T13:12:40.065063+0000","last_fullsized":"2024-04-13T13:12:40.065063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T18:23:45.790258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.066226+0000","last_change":"2024-04-13T13:12:40.061093+0000","last_active":"2024-04-13T13:12:40.066226+0000","last_peered":"2024-04-13T13:12:40.066226+0000","last_clean":"2024-04-13T13:12:40.066226+0000","last_became_active":"2024-04-13T13:12:40.060813+0000","last_became_peered":"2024-04-13T13:12:40.060813+0000","last_unstale":"2024-04-13T13:12:40.066226+0000","last_undegraded":"2024-04-13T13:12:40.066226+0000","last_fullsized":"2024-04-13T13:12:40.066226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:27:19.714458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T13:12:39.051206+0000","last_change":"2024-04-13T13:12:35.672780+0000","last_active":"2024-04-13T13:12:39.051206+0000","last_peered":"2024-04-13T13:12:39.051206+0000","last_clean":"2024-04-13T13:12:39.051206+0000","last_became_active":"2024-04-13T13:12:35.671780+0000","last_became_peered":"2024-04-13T13:12:35.671780+0000","last_unstale":"2024-04-13T13:12:39.051206+0000","last_undegraded":"2024-04-13T13:12:39.051206+0000","last_fullsized":"2024-04-13T13:12:39.051206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:34.032923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:34.032923+0000","last_clean_scrub_stamp":"2024-04-13T13:12:34.032923+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T21:07:17.372667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T13:12:43.161 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-13T13:12:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:43.439 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-04-13T13:12:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-13T13:12:43.224079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.178175"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.068929+0000","last_change":"2024-04-13T13:12:40.063803+0000","last_active":"2024-04-13T13:12:40.068929+0000","last_peered":"2024-04-13T13:12:40.068929+0000","last_clean":"2024-04-13T13:12:40.068929+0000","last_became_active":"2024-04-13T13:12:40.063533+0000","last_became_peered":"2024-04-13T13:12:40.063533+0000","last_unstale":"2024-04-13T13:12:40.068929+0000","last_undegraded":"2024-04-13T13:12:40.068929+0000","last_fullsized":"2024-04-13T13:12:40.068929+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T20:58:21.866563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.068914+0000","last_change":"2024-04-13T13:12:40.063540+0000","last_active":"2024-04-13T13:12:40.068914+0000","last_peered":"2024-04-13T13:12:40.068914+0000","last_clean":"2024-04-13T13:12:40.068914+0000","last_became_active":"2024-04-13T13:12:40.063264+0000","last_became_peered":"2024-04-13T13:12:40.063264+0000","last_unstale":"2024-04-13T13:12:40.068914+0000","last_undegraded":"2024-04-13T13:12:40.068914+0000","last_fullsized":"2024-04-13T13:12:40.068914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T20:41:39.391010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.068921+0000","last_change":"2024-04-13T13:12:40.063758+0000","last_active":"2024-04-13T13:12:40.068921+0000","last_peered":"2024-04-13T13:12:40.068921+0000","last_clean":"2024-04-13T13:12:40.068921+0000","last_became_active":"2024-04-13T13:12:40.063310+0000","last_became_peered":"2024-04-13T13:12:40.063310+0000","last_unstale":"2024-04-13T13:12:40.068921+0000","last_undegraded":"2024-04-13T13:12:40.068921+0000","last_fullsized":"2024-04-13T13:12:40.068921+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:58:37.147448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.068993+0000","last_change":"2024-04-13T13:12:40.064229+0000","last_active":"2024-04-13T13:12:40.068993+0000","last_peered":"2024-04-13T13:12:40.068993+0000","last_clean":"2024-04-13T13:12:40.068993+0000","last_became_active":"2024-04-13T13:12:40.063910+0000","last_became_peered":"2024-04-13T13:12:40.063910+0000","last_unstale":"2024-04-13T13:12:40.068993+0000","last_undegraded":"2024-04-13T13:12:40.068993+0000","last_fullsized":"2024-04-13T13:12:40.068993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T00:43:25.158364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-13T13:12:41.064368+0000","last_change":"2024-04-13T13:12:40.064583+0000","last_active":"2024-04-13T13:12:41.064368+0000","last_peered":"2024-04-13T13:12:41.064368+0000","last_clean":"2024-04-13T13:12:41.064368+0000","last_became_active":"2024-04-13T13:12:40.064295+0000","last_became_peered":"2024-04-13T13:12:40.064295+0000","last_unstale":"2024-04-13T13:12:41.064368+0000","last_undegraded":"2024-04-13T13:12:41.064368+0000","last_fullsized":"2024-04-13T13:12:41.064368+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T00:46:27.385820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.065070+0000","last_change":"2024-04-13T13:12:40.063508+0000","last_active":"2024-04-13T13:12:40.065070+0000","last_peered":"2024-04-13T13:12:40.065070+0000","last_clean":"2024-04-13T13:12:40.065070+0000","last_became_active":"2024-04-13T13:12:40.063218+0000","last_became_peered":"2024-04-13T13:12:40.063218+0000","last_unstale":"2024-04-13T13:12:40.065070+0000","last_undegraded":"2024-04-13T13:12:40.065070+0000","last_fullsized":"2024-04-13T13:12:40.065070+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:15:41.174354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.065063+0000","last_change":"2024-04-13T13:12:40.063540+0000","last_active":"2024-04-13T13:12:40.065063+0000","last_peered":"2024-04-13T13:12:40.065063+0000","last_clean":"2024-04-13T13:12:40.065063+0000","last_became_active":"2024-04-13T13:12:40.063266+0000","last_became_peered":"2024-04-13T13:12:40.063266+0000","last_unstale":"2024-04-13T13:12:40.065063+0000","last_undegraded":"2024-04-13T13:12:40.065063+0000","last_fullsized":"2024-04-13T13:12:40.065063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T18:23:45.790258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-13T13:12:40.066226+0000","last_change":"2024-04-13T13:12:40.061093+0000","last_active":"2024-04-13T13:12:40.066226+0000","last_peered":"2024-04-13T13:12:40.066226+0000","last_clean":"2024-04-13T13:12:40.066226+0000","last_became_active":"2024-04-13T13:12:40.060813+0000","last_became_peered":"2024-04-13T13:12:40.060813+0000","last_unstale":"2024-04-13T13:12:40.066226+0000","last_undegraded":"2024-04-13T13:12:40.066226+0000","last_fullsized":"2024-04-13T13:12:40.066226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:39.043983+0000","last_clean_scrub_stamp":"2024-04-13T13:12:39.043983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T22:27:19.714458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-13T13:12:39.051206+0000","last_change":"2024-04-13T13:12:35.672780+0000","last_active":"2024-04-13T13:12:39.051206+0000","last_peered":"2024-04-13T13:12:39.051206+0000","last_clean":"2024-04-13T13:12:39.051206+0000","last_became_active":"2024-04-13T13:12:35.671780+0000","last_became_peered":"2024-04-13T13:12:35.671780+0000","last_unstale":"2024-04-13T13:12:39.051206+0000","last_undegraded":"2024-04-13T13:12:39.051206+0000","last_fullsized":"2024-04-13T13:12:39.051206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T13:12:34.032923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T13:12:34.032923+0000","last_clean_scrub_stamp":"2024-04-13T13:12:34.032923+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T21:07:17.372667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T13:12:43.451 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-13T13:12:43.452 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-04-13T13:12:43.452 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-04-13T13:12:43.452 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-04-13T13:12:43.763 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-13T13:12:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-13T13:12:43.775 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-04-13T13:12:43.775 INFO:teuthology.run_tasks:Running task exec... 2024-04-13T13:12:43.785 INFO:teuthology.task.exec:Executing custom commands... 2024-04-13T13:12:43.785 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T13:12:43.785 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:44.146 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror 2024-04-13T13:12:44.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:44.520 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.0 2024-04-13T13:12:44.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:44.892 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.1 2024-04-13T13:12:44.905 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:45.264 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.2 2024-04-13T13:12:45.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.3 2024-04-13T13:12:45.656 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:46.015 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror 2024-04-13T13:12:46.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:46.387 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.0 2024-04-13T13:12:46.400 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.1 2024-04-13T13:12:46.773 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:47.131 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.2 2024-04-13T13:12:47.145 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-13T13:12:47.508 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.3 2024-04-13T13:12:47.521 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-13T13:12:47.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-13T13:12:47.540 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-13T13:12:47.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-13T13:12:47.543 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-13T13:12:47.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-13T13:12:47.552 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-13T13:12:47.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-13T13:12:47.554 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-13T13:12:47.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-13T13:12:47.563 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-13T13:12:47.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-13T13:12:47.565 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-13T13:12:47.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-13T13:12:47.574 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-13T13:12:47.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-13T13:12:47.576 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-13T13:12:47.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-13T13:12:47.585 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-13T13:12:47.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-13T13:12:47.587 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-13T13:12:47.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-13T13:12:47.596 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-13T13:12:47.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-13T13:12:47.598 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-13T13:12:47.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-13T13:12:47.607 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-13T13:12:47.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-13T13:12:47.611 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-13T13:12:47.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-13T13:12:47.621 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-13T13:12:47.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-13T13:12:47.625 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-13T13:12:47.637 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1713013967 2024-04-13T13:12:47.637 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-13T13:12:47.637 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-13T13:12:47.648 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1713013967 2024-04-13T13:12:47.648 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-13T13:12:47.648 INFO:teuthology.run_tasks:Running task workunit... 2024-04-13T13:12:47.659 INFO:tasks.workunit:Pulling workunits from ref 4f70a9a90881107996c8b7a3e89ba8b32624713c 2024-04-13T13:12:47.660 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-13T13:12:47.660 DEBUG:teuthology.orchestra.run.smithi122:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-13T13:12:47.660 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-04-13T13:12:47.661 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-04-13T13:12:47.661 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-04-13T13:12:47.661 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-04-13T13:12:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T13:12:47.676 INFO:teuthology.orchestra.run.smithi122.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-04-13T13:12:47.676 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-13T13:12:47.730 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-13T13:12:47.730 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-04-13T13:12:47.784 INFO:tasks.workunit:timeout=6h 2024-04-13T13:12:47.784 INFO:tasks.workunit:cleanup=True 2024-04-13T13:12:47.785 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 4f70a9a90881107996c8b7a3e89ba8b32624713c 2024-04-13T13:12:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-04-13T13:13:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Note: switching to '4f70a9a90881107996c8b7a3e89ba8b32624713c'. 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:state without impacting any branches by switching back to a branch. 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: git switch -c 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Or undo this operation with: 2024-04-13T13:13:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:13:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: git switch - 2024-04-13T13:13:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:13:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-13T13:13:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:13:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:HEAD is now at 4f70a9a9088 Merge pull request #56780 from zdover23/wip-doc-2024-04-09-backport-56754-to-reef 2024-04-13T13:13:54.904 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-04-13T13:13:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-13T13:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-13T13:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-13T13:13:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-13T13:13:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-13T13:13:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-13T13:13:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-13T13:13:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-13T13:13:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-13T13:13:55.892 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-13T13:13:55.892 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-04-13T13:13:55.948 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-04-13T13:13:55.950 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-04-13T13:13:55.950 DEBUG:teuthology.orchestra.run.smithi122:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f70a9a90881107996c8b7a3e89ba8b32624713c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-13T13:13:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ IMAGE_COUNT=50 2024-04-13T13:13:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ export LOCKDEP=0 2024-04-13T13:13:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ LOCKDEP=0 2024-04-13T13:13:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-13T13:13:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-04-13T13:13:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ type xmlstarlet 2024-04-13T13:13:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ XMLSTARLET=xmlstarlet 2024-04-13T13:13:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ RBD_MIRROR_INSTANCES=4 2024-04-13T13:13:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CLUSTER1=cluster1 2024-04-13T13:13:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CLUSTER2=cluster2 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ PEER_CLUSTER_SUFFIX= 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ POOL=mirror 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ PARENT_POOL=mirror_parent 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ NS1=ns1 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ NS2=ns2 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ TEMPDIR= 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ID=mirror 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ MIRROR_POOL_MODE=image 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ MIRROR_IMAGE_MODE=snapshot 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS='--id mirror' 2024-04-13T13:13:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ LAST_MIRROR_INSTANCE=3 2024-04-13T13:13:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-13T13:13:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-04-13T13:13:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-04-13T13:13:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_BIN=. 2024-04-13T13:13:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_SRC=. 2024-04-13T13:13:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' -e CMakeCache.txt ']' 2024-04-13T13:13:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 -gt 0 ']' 2024-04-13T13:13:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ setup 2024-04-13T13:13:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local c 2024-04-13T13:13:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-04-13T13:13:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ setup_tempdir 2024-04-13T13:13:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:13:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mktemp -d 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ TEMPDIR=/tmp/tmp.6pyizPAJcx 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -z 1 ']' 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ setup_pools cluster1 cluster2 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster2 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mon_map_file 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mon_addr 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local admin_key_file 2024-04-13T13:13:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid 2024-04-13T13:13:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:13:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-04-13T13:13:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'mirror' already exists 2024-04-13T13:13:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:13:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-04-13T13:13:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'mirror_parent' already exists 2024-04-13T13:13:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:13:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 pool init mirror 2024-04-13T13:14:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:14:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-04-13T13:14:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:14:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-04-13T13:14:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-04-13T13:14:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-04-13T13:14:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-04-13T13:14:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-04-13T13:14:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-04-13T13:14:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -z ']' 2024-04-13T13:14:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -z ']' 2024-04-13T13:14:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster1 mirror cluster2 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster2 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster2 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 16 32 2024-04-13T13:14:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:14:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-04-13T13:14:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=a2d8dee1-b6c0-44ee-9be5-600f2a9f2571 2024-04-13T13:14:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2024-04-13T13:14:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:14:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 0 -eq 17 ']' 2024-04-13T13:14:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2024-04-13T13:14:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster2 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster2 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 16 32 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:14:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-04-13T13:14:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=963bf1d1-056a-441a-b7b6-480b4924a6c2 2024-04-13T13:14:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2024-04-13T13:14:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:14:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 0 -eq 17 ']' 2024-04-13T13:14:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2024-04-13T13:14:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ setup_pools cluster2 cluster1 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster1 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mon_map_file 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mon_addr 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local admin_key_file 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:14:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-04-13T13:14:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'mirror' already exists 2024-04-13T13:14:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:14:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-04-13T13:14:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'mirror_parent' already exists 2024-04-13T13:14:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:14:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 pool init mirror 2024-04-13T13:14:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:14:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-04-13T13:14:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:14:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-04-13T13:14:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-04-13T13:14:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-04-13T13:14:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-04-13T13:14:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-04-13T13:14:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -z ']' 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -z ']' 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster2 mirror cluster1 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster1 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster1 2024-04-13T13:14:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2024-04-13T13:14:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2024-04-13T13:14:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2024-04-13T13:14:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 16 32 2024-04-13T13:14:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:14:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-04-13T13:14:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=02d58163-94d8-4e06-8224-e3a9ea90d177 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 0 -eq 17 ']' 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster1 2024-04-13T13:14:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster1 2024-04-13T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2024-04-13T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2024-04-13T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2024-04-13T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 16 32 2024-04-13T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-04-13T13:14:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=188eb3f2-be06-4e17-ae31-62e0820f8660 2024-04-13T13:14:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2024-04-13T13:14:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:14:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 0 -eq 17 ']' 2024-04-13T13:14:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2024-04-13T13:14:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirrors cluster1 2024-04-13T13:14:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 0 3 2024-04-13T13:14:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:0 2024-04-13T13:14:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2024-04-13T13:14:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2024-04-13T13:14:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:14:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:1 2024-04-13T13:14:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:14:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:2 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:3 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2024-04-13T13:14:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirrors cluster2 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 0 3 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster2:0 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2024-04-13T13:14:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 0 = cluster2:0 ']' 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster2 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T13:14:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T13:14:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster2:1 2024-04-13T13:14:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2024-04-13T13:14:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 1 = cluster2:1 ']' 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster2 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster2:2 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 2 = cluster2:2 ']' 2024-04-13T13:14:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2024-04-13T13:14:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster2 2024-04-13T13:14:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2024-04-13T13:14:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:14:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T13:14:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T13:14:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster2:3 2024-04-13T13:14:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2024-04-13T13:14:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2024-04-13T13:14:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-04-13T13:14:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 3 = cluster2:3 ']' 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster2 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:14:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.6pyizPAJcx/rbd-mirror.test.log 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ date '+%F %T' 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13 13:14:11 TEST: add image and test replay after client crashes 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13 13:14:11 TEST: add image and test replay after client crashes 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=test 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test 512M 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 512M ']' 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=512M 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:14:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2024-04-13T13:14:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:14:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-04-13T13:14:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:14:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-04-13T13:14:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:14:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-04-13T13:14:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:14:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:14:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:14:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:14:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a4c8bd4265 2024-04-13T13:14:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a4c8bd4265 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:11 2024 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:14:11 2024 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:11 2024 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:14:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:14:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:14:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:14:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:14:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:14:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:14:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:14:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:14:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:14:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:14:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:14:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:14:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:14:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:14:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:14:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:14:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:14:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:14:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:14:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:14:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:14:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:14:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:14:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:14:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:14:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:14:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:14:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:14:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:14:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:14:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:14:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-13T13:14:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 1 2024-04-13T13:14:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:14:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:14:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:14:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:14:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:14:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:14:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:14:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:14:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:14:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:14:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:14:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:14:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:14:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:14:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:14:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:14:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:14:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:14:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-13T13:14:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:14:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:14:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:14:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:14:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:14:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-13T13:14:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 2 2024-04-13T13:14:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:14:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:14:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:14:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:14:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:14:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:14:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:14:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:14:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:14:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:14:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:14:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:14:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:14:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:14:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:14:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:14:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:14:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:14:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:14:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:14:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:14:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:14:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:14:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:14:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-13T13:14:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:14:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:14:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:14:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:14:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:14:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-13T13:14:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:14:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:14:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:14:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 4 2024-04-13T13:14:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:14:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:14:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:14:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:14:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:14:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:14:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:14:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:14:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:14:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:14:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:14:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:14:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:14:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:14:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:14:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name= 2024-04-13T13:14:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 10 2024-04-13T13:14:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:14:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:14:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:14:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=6 2024-04-13T13:14:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:14:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:14:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 33711104 bytes 2024-04-13T13:14:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 76185600 bytes 2024-04-13T13:14:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 163475456 bytes 2024-04-13T13:14:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 198998016 bytes 2024-04-13T13:14:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 240228352 bytes 2024-04-13T13:14:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 262350848 bytes 2024-04-13T13:14:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 370495488 bytes 2024-04-13T13:14:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 398067712 bytes 2024-04-13T13:14:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 452378624 bytes 2024-04-13T13:14:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 529081344 bytes 2024-04-13T13:14:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 543073280 bytes 2024-04-13T13:14:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 546233344 bytes 2024-04-13T13:14:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 569521152 bytes 2024-04-13T13:14:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 622687232 bytes 2024-04-13T13:14:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 648778752 bytes 2024-04-13T13:14:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 700429312 bytes 2024-04-13T13:14:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 723398656 bytes 2024-04-13T13:14:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 742133760 bytes 2024-04-13T13:14:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 752160768 bytes 2024-04-13T13:14:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 779525120 bytes 2024-04-13T13:14:24.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 841624576 bytes 2024-04-13T13:14:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 939682816 bytes 2024-04-13T13:14:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 949038080 bytes 2024-04-13T13:14:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1017039872 bytes 2024-04-13T13:14:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1063023616 bytes 2024-04-13T13:14:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1149390848 bytes 2024-04-13T13:14:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1180901376 bytes 2024-04-13T13:14:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1182888960 bytes 2024-04-13T13:14:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1261122560 bytes 2024-04-13T13:14:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1290459136 bytes 2024-04-13T13:14:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1332093952 bytes 2024-04-13T13:14:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1421555712 bytes 2024-04-13T13:14:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1480636416 bytes 2024-04-13T13:14:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1502107648 bytes 2024-04-13T13:14:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1573011456 bytes 2024-04-13T13:14:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1633916928 bytes 2024-04-13T13:14:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1634531328 bytes 2024-04-13T13:14:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1697727488 bytes 2024-04-13T13:14:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1748139008 bytes 2024-04-13T13:14:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1789252608 bytes 2024-04-13T13:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1820907520 bytes 2024-04-13T13:14:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:14:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:14:28.662 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9161504820896161) > weight (0.25) 2024-04-13T13:14:28.662 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.41656327189308573) > weight (0.25) 2024-04-13T13:14:28.662 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2531658973741786) > weight (0.25) 2024-04-13T13:14:28.662 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-13T13:14:28.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-13T13:14:28.663 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-13T13:14:28.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi122.stderr:2024-04-13T13:14:28.663+0000 7fa058df9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 39650) UID: 1000 2024-04-13T13:14:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:14:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:14:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:14:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:14:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:14:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:14:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:14:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:14:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:14:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:14:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:14:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13T13:14:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:14:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:14:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:14:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:14:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: ' 2024-04-13T13:14:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:14:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2024-04-13T13:14:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2024-04-13T13:14:29.662 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4292275868965655) > weight (0.25) 2024-04-13T13:14:29.662 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-13T13:14:29.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-13T13:14:29.663 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-13T13:14:29.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi122.stderr:2024-04-13T13:14:29.663+0000 7f10881f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39686) UID: 1000 2024-04-13T13:14:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:14:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:14:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: ' 2024-04-13T13:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:14:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2024-04-13T13:14:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2024-04-13T13:14:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:14:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:14:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:14:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: ' 2024-04-13T13:14:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:14:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2024-04-13T13:14:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2024-04-13T13:14:35.664 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-13T13:14:35.664 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:14:35.665 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-13T13:14:35.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-13T13:14:35.665 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-13T13:14:35.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-13T13:14:35.668 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-04-13T13:14:36.663 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-13T13:14:36.664 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:14:36.664 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-13T13:14:36.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-13T13:14:36.664 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-13T13:14:36.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-13T13:14:36.667 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-04-13T13:14:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:14:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:14:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:14:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:14:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:14:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:14:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:14:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:14:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: ' 2024-04-13T13:14:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:14:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2024-04-13T13:14:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:14:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:14:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:14:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:14:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:14:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:14:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713014052,"remote_snapshot_timestamp":1713014052,"replay_state":"idle"} 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:14:47 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:14:47' 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:14:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:14:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:14:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713014052,"remote_snapshot_timestamp":1713014052,"replay_state":"idle"} 2024-04-13T13:14:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:14:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:14:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:14:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:14:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:14:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:14:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:14:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:14:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:14:47' 2024-04-13T13:14:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:14:47' 2024-04-13T13:14:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:14:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:14:47' 2024-04-13T13:14:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:14:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:14:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:14:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42154263mirror.0smithi12218.2.2-1187-gd540ebactrueOK42194267mirror.2smithi12218.2.2-1187-gd540ebacfalseOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713014052,"remote_snapshot_timestamp":1713014052,"replay_state":"idle"}42154263mirror.0smithi1222024-04-13 13:14:47cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:14:47' 2024-04-13T13:14:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:14:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:14:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:14:47' 2024-04-13T13:14:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:14:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:14:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:14:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:14:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:14:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap1 2024-04-13T13:14:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap1 2024-04-13T13:14:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap1 2024-04-13T13:14:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:14:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:14:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:14:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:14:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:14:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:14:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:14:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:14:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:14:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:14:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:14:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:14:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:14:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:14:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:14:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:14:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:14:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:14:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:14:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:14:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:14:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:14:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:14:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:14:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:14:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:14:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:14:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:14:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:14:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:15:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 6 2024-04-13T13:15:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:15:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:15:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:15:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:15:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:15:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=6 2024-04-13T13:15:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:15:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=4 2024-04-13T13:15:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 6 = 4 2024-04-13T13:15:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:15:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:15:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:15:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:15:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 6 = '' 2024-04-13T13:15:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:15:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:15:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:15:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:15:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:15:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 6 = '' 2024-04-13T13:15:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:15:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=6 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 6 = 6 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap1 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:15:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:15:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:15:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:15:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:15:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:15:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:15:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:15:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:15:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:15:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap1 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap1 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:15:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:15:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:15:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:15:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:15:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:15:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:15:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:15:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:15:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap1-clean 2024-04-13T13:15:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-04-13T13:15:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:15:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap1-clean 2024-04-13T13:15:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:15:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:15:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:15:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:15:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:15:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:15:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:15:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:15:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 68370432 bytes 2024-04-13T13:15:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 96265216 bytes 2024-04-13T13:15:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 119568384 bytes 2024-04-13T13:15:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 120175616 bytes 2024-04-13T13:15:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 147545088 bytes 2024-04-13T13:15:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 162262016 bytes 2024-04-13T13:15:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 226170880 bytes 2024-04-13T13:15:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 234161152 bytes 2024-04-13T13:15:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 274859008 bytes 2024-04-13T13:15:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 275466240 bytes 2024-04-13T13:15:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 335239168 bytes 2024-04-13T13:15:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 335910912 bytes 2024-04-13T13:15:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 359244800 bytes 2024-04-13T13:15:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 371652608 bytes 2024-04-13T13:15:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 456527872 bytes 2024-04-13T13:15:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 491403264 bytes 2024-04-13T13:15:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 501550080 bytes 2024-04-13T13:15:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 538160128 bytes 2024-04-13T13:15:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 564222976 bytes 2024-04-13T13:15:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 576420864 bytes 2024-04-13T13:15:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 624925696 bytes 2024-04-13T13:15:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 728302592 bytes 2024-04-13T13:15:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 736671744 bytes 2024-04-13T13:15:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 756943872 bytes 2024-04-13T13:15:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 807817216 bytes 2024-04-13T13:15:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 863029248 bytes 2024-04-13T13:15:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 901583872 bytes 2024-04-13T13:15:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 933137408 bytes 2024-04-13T13:15:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 973064192 bytes 2024-04-13T13:15:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1027894272 bytes 2024-04-13T13:15:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1109087232 bytes 2024-04-13T13:15:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1137118208 bytes 2024-04-13T13:15:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1152797696 bytes 2024-04-13T13:15:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1183951872 bytes 2024-04-13T13:15:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1237641216 bytes 2024-04-13T13:15:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1267558400 bytes 2024-04-13T13:15:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1287866368 bytes 2024-04-13T13:15:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1331974144 bytes 2024-04-13T13:15:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1343816704 bytes 2024-04-13T13:15:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1344418816 bytes 2024-04-13T13:15:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1390679040 bytes 2024-04-13T13:15:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1444923392 bytes 2024-04-13T13:15:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1471145984 bytes 2024-04-13T13:15:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1522368512 bytes 2024-04-13T13:15:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1522998272 bytes 2024-04-13T13:15:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1533696000 bytes 2024-04-13T13:15:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1585232896 bytes 2024-04-13T13:15:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1689232384 bytes 2024-04-13T13:15:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1757820928 bytes 2024-04-13T13:15:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1758416896 bytes 2024-04-13T13:15:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1792183296 bytes 2024-04-13T13:15:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1837648896 bytes 2024-04-13T13:15:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 1867583488 bytes 2024-04-13T13:15:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 1894994944 bytes 2024-04-13T13:15:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 1947957248 bytes 2024-04-13T13:15:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 1985677312 bytes 2024-04-13T13:15:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2012525568 bytes 2024-04-13T13:15:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2106871808 bytes 2024-04-13T13:15:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2130349056 bytes 2024-04-13T13:15:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2174475264 bytes 2024-04-13T13:15:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2203297792 bytes 2024-04-13T13:15:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2249241600 bytes 2024-04-13T13:15:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2316139520 bytes 2024-04-13T13:15:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2316785664 bytes 2024-04-13T13:15:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2364729344 bytes 2024-04-13T13:15:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2418437120 bytes 2024-04-13T13:15:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2445386752 bytes 2024-04-13T13:15:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2468343808 bytes 2024-04-13T13:15:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2506736640 bytes 2024-04-13T13:15:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2507371520 bytes 2024-04-13T13:15:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2527001600 bytes 2024-04-13T13:15:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2544203776 bytes 2024-04-13T13:15:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2561671168 bytes 2024-04-13T13:15:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 2651781120 bytes 2024-04-13T13:15:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 2671378432 bytes 2024-04-13T13:15:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 2701506560 bytes 2024-04-13T13:15:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 2712722432 bytes 2024-04-13T13:15:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7800 IOs, 2777186304 bytes 2024-04-13T13:15:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7900 IOs, 2816533504 bytes 2024-04-13T13:15:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8000 IOs, 2831962112 bytes 2024-04-13T13:15:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8100 IOs, 2857187328 bytes 2024-04-13T13:15:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8200 IOs, 2887465984 bytes 2024-04-13T13:15:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8300 IOs, 2926104576 bytes 2024-04-13T13:15:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8400 IOs, 2994640896 bytes 2024-04-13T13:15:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8500 IOs, 2995273728 bytes 2024-04-13T13:15:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8600 IOs, 3025428480 bytes 2024-04-13T13:15:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8700 IOs, 3118958592 bytes 2024-04-13T13:15:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8800 IOs, 3210527744 bytes 2024-04-13T13:15:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8900 IOs, 3276285952 bytes 2024-04-13T13:15:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:15:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:15:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:15:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:15:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:15:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:15:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":17138585.6,"bytes_per_snapshot":257078784.0,"last_snapshot_bytes":514157568,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014100,"remote_snapshot_timestamp":1713014100,"replay_state":"idle"} 2024-04-13T13:15:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2024-04-13T13:15:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:15:17 2024-04-13T13:15:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:15:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:15:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:15:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:15:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:15:17' 2024-04-13T13:15:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:15:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:15:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:15:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":17138585.6,"bytes_per_snapshot":257078784.0,"last_snapshot_bytes":514157568,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014100,"remote_snapshot_timestamp":1713014100,"replay_state":"idle"} 2024-04-13T13:15:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:15:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:15:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:15:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:15:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:15:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:15:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:15:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:15:17' 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:15:17' 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:15:17' 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:15:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:15:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42154263mirror.0smithi12218.2.2-1187-gd540ebactrueOK42194267mirror.2smithi12218.2.2-1187-gd540ebacfalseOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":17138585.6,"bytes_per_snapshot":257078784.0,"last_snapshot_bytes":514157568,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014100,"remote_snapshot_timestamp":1713014100,"replay_state":"idle"}42154263mirror.0smithi1222024-04-13 13:15:17cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:15:17' 2024-04-13T13:15:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:15:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:15:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:15:17' 2024-04-13T13:15:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:15:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:15:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:15:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:15:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap2 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap2 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2024-04-13T13:15:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:15:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:15:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:15:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:15:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:15:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:15:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:15:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:15:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:15:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:15:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:15:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:15:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:15:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:15:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:15:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:15:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:15:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:15:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:15:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:15:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:15:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:15:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:15:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:15:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:15:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:15:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:15:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:15:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:15:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:15:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:15:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:15:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:15:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:15:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:15:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:15:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:15:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:15:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:15:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 9 2024-04-13T13:15:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:15:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=9 2024-04-13T13:15:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:15:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:15:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:15:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:15:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:15:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:15:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:15:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:15:53.668 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-13T13:15:53.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-13T13:15:53.669 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-13T13:15:53.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi122.stderr:2024-04-13T13:15:53.670+0000 7fbca41f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39600) UID: 1000 2024-04-13T13:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:15:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:15:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:15:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:15:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:15:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:15:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:15:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:15:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:15:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:15:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:15:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2024-04-13T13:15:56.670 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-13T13:15:56.670 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:15:56.670 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-13T13:15:56.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-13T13:15:56.671 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-13T13:15:56.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-13T13:15:56.674 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-04-13T13:15:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:15:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:15:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:15:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:15:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:15:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2024-04-13T13:15:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:15:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:15:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:15:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:15:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:15:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:15:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:15:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:15:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:15:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:15:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:15:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:15:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:15:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:15:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2024-04-13T13:16:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:16:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:16:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:16:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:16:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:16:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:16:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:16:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:16:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:16:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:16:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2024-04-13T13:16:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:16:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:16:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:16:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2024-04-13T13:16:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:16:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:16:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:16:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:16:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:16:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:16:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:16:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:16:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:16:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:16:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2024-04-13T13:16:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:16:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:16:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:16:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:16:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:16:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:16:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = '' 2024-04-13T13:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:16:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 16 2024-04-13T13:16:25.668 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5233039373456133) > weight (0.25) 2024-04-13T13:16:25.669 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4236935649020901) > weight (0.25) 2024-04-13T13:16:25.669 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3359164726397966) > weight (0.25) 2024-04-13T13:16:25.669 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-13T13:16:25.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-13T13:16:25.669 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-13T13:16:25.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi122.stderr:2024-04-13T13:16:25.670+0000 7f659c3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 39719) UID: 1000 2024-04-13T13:16:33.670 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-13T13:16:33.670 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:16:33.671 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-13T13:16:33.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-13T13:16:33.671 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-13T13:16:33.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-13T13:16:33.675 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-04-13T13:16:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:16:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:16:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:16:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=9 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 9 = 9 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:16:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:16:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-04-13T13:16:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:16:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:16:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:16:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 4 2024-04-13T13:16:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:16:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:16:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap1-clean ']' 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap1-clean 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:16:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:16:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:16:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:16:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap2 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:16:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:16:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:16:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:16:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:16:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap2-clean 2024-04-13T13:16:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-04-13T13:16:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:16:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2-clean 2024-04-13T13:16:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=8 2024-04-13T13:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:16:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 54791168 bytes 2024-04-13T13:16:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 102892544 bytes 2024-04-13T13:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 211490816 bytes 2024-04-13T13:16:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 249656320 bytes 2024-04-13T13:16:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 286315520 bytes 2024-04-13T13:16:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 315162624 bytes 2024-04-13T13:16:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 343071744 bytes 2024-04-13T13:16:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 368287744 bytes 2024-04-13T13:16:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 368878592 bytes 2024-04-13T13:16:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 416918528 bytes 2024-04-13T13:16:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 490025984 bytes 2024-04-13T13:16:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 493618176 bytes 2024-04-13T13:16:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 506030080 bytes 2024-04-13T13:16:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 596882432 bytes 2024-04-13T13:16:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 658130944 bytes 2024-04-13T13:16:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 680679424 bytes 2024-04-13T13:16:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 708384768 bytes 2024-04-13T13:16:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 761992192 bytes 2024-04-13T13:16:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 812053504 bytes 2024-04-13T13:16:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 823744512 bytes 2024-04-13T13:16:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 878441472 bytes 2024-04-13T13:16:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 920033280 bytes 2024-04-13T13:16:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 948017152 bytes 2024-04-13T13:16:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 966242304 bytes 2024-04-13T13:16:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 988297216 bytes 2024-04-13T13:16:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1039898624 bytes 2024-04-13T13:16:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1077337088 bytes 2024-04-13T13:16:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1130029056 bytes 2024-04-13T13:16:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1176295424 bytes 2024-04-13T13:16:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1185610752 bytes 2024-04-13T13:16:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1220024320 bytes 2024-04-13T13:16:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1262105600 bytes 2024-04-13T13:16:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1308901376 bytes 2024-04-13T13:16:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1358688256 bytes 2024-04-13T13:16:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1392348160 bytes 2024-04-13T13:16:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1430977536 bytes 2024-04-13T13:16:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1449182208 bytes 2024-04-13T13:16:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1497604096 bytes 2024-04-13T13:16:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1553582080 bytes 2024-04-13T13:16:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1605549056 bytes 2024-04-13T13:16:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1656334336 bytes 2024-04-13T13:16:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1705817088 bytes 2024-04-13T13:16:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1751101440 bytes 2024-04-13T13:16:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1768126464 bytes 2024-04-13T13:16:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1805299712 bytes 2024-04-13T13:16:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1835692032 bytes 2024-04-13T13:16:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1898323968 bytes 2024-04-13T13:16:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1985132544 bytes 2024-04-13T13:16:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2004526080 bytes 2024-04-13T13:16:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2035536896 bytes 2024-04-13T13:16:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2076853248 bytes 2024-04-13T13:16:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2110457856 bytes 2024-04-13T13:16:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2191583232 bytes 2024-04-13T13:16:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2273700864 bytes 2024-04-13T13:16:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:16:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:16:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:16:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:16:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:16:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:16:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:16:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:16:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:16:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:16:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:16:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:16:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:16:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:16:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:16:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:16:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:16:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:16:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":26797056.0,"bytes_per_snapshot":535941120.0,"last_snapshot_bytes":535941120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014151,"remote_snapshot_timestamp":1713014151,"replay_state":"idle"} 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:16:47 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:16:47' 2024-04-13T13:16:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:16:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:16:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:16:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":26797056.0,"bytes_per_snapshot":535941120.0,"last_snapshot_bytes":535941120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014151,"remote_snapshot_timestamp":1713014151,"replay_state":"idle"} 2024-04-13T13:16:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:16:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:16:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:16:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:16:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:16:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:16:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:16:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:16:47' 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:16:47' 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:16:47' 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:16:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42194267mirror.2smithi12218.2.2-1187-gd540ebactrueOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOK43774380mirror.0smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":26797056.0,"bytes_per_snapshot":535941120.0,"last_snapshot_bytes":535941120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014151,"remote_snapshot_timestamp":1713014151,"replay_state":"idle"}42144264mirror.1smithi1222024-04-13 13:16:47cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:16:47' 2024-04-13T13:16:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:16:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:16:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:16:47' 2024-04-13T13:16:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:16:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:16:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:16:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:16:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap3 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap3 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap3 2024-04-13T13:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:17:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:17:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:17:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:17:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:17:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:17:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:17:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:17:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:17:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:17:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:17:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:17:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:17:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:17:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:17:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:17:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:17:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:17:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:17:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:17:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:17:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:17:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:17:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:17:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:17:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:17:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:17:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:17:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:17:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:17:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:17:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:17:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:17:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:17:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:17:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:17:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:17:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:17:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:17:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:17:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:17:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:17:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:17:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:17:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:17:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:17:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:17:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:17:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 13 2024-04-13T13:17:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:17:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:17:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:17:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:17:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:17:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:17:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=13 2024-04-13T13:17:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:17:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:17:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:17:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:17:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:17:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:17:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:17:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:17:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:17:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:17:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:17:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 13 = '' 2024-04-13T13:17:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:17:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:17:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:17:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:17:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:17:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:17:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:17:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:17:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:17:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:17:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 13 = '' 2024-04-13T13:17:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:17:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:17:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:17:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:17:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:17:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:17:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:17:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:17:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:17:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:17:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 13 = '' 2024-04-13T13:17:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:17:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:17:29.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7681724116604935) > weight (0.25) 2024-04-13T13:17:29.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5493607031202439) > weight (0.25) 2024-04-13T13:17:29.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.35590922748195575) > weight (0.25) 2024-04-13T13:17:29.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.987494606023278) > weight (0.25) 2024-04-13T13:17:29.675 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-04-13T13:17:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:17:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:17:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:17:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:17:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=13 2024-04-13T13:17:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 13 = 13 2024-04-13T13:17:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:17:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-04-13T13:17:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap3 2024-04-13T13:17:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:17:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:17:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:17:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 6 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:17:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:17:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:16:41 2024 2024-04-13T13:17:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:17:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:17:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:17:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:17:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:17:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:17:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:17:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap2-clean ']' 2024-04-13T13:17:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-13T13:17:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2-clean 2024-04-13T13:17:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:17:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:17:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:17:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:17:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:17:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:17:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap3 2024-04-13T13:17:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap3 2024-04-13T13:17:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:17:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:17:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:17:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:17:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:17:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap3-clean 2024-04-13T13:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-04-13T13:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap3-clean 2024-04-13T13:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:17:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:17:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:17:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:17:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:17:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=10 2024-04-13T13:17:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:17:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:17:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 26441728 bytes 2024-04-13T13:17:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 44256256 bytes 2024-04-13T13:17:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 100972544 bytes 2024-04-13T13:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 125660160 bytes 2024-04-13T13:17:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 127499264 bytes 2024-04-13T13:17:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 194329600 bytes 2024-04-13T13:17:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 247322624 bytes 2024-04-13T13:17:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 307058688 bytes 2024-04-13T13:17:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 337688576 bytes 2024-04-13T13:17:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 388260864 bytes 2024-04-13T13:17:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 433250304 bytes 2024-04-13T13:17:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 446615552 bytes 2024-04-13T13:17:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 452265984 bytes 2024-04-13T13:17:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 453066752 bytes 2024-04-13T13:17:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 493176832 bytes 2024-04-13T13:17:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 512793600 bytes 2024-04-13T13:17:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 554335232 bytes 2024-04-13T13:17:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 612828160 bytes 2024-04-13T13:17:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 651309056 bytes 2024-04-13T13:17:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 689537024 bytes 2024-04-13T13:17:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 728328192 bytes 2024-04-13T13:17:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 802319360 bytes 2024-04-13T13:17:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 835107840 bytes 2024-04-13T13:17:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 887133184 bytes 2024-04-13T13:17:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 938616832 bytes 2024-04-13T13:17:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 956421120 bytes 2024-04-13T13:17:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1002682368 bytes 2024-04-13T13:17:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1052667904 bytes 2024-04-13T13:17:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1077620736 bytes 2024-04-13T13:17:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1161550848 bytes 2024-04-13T13:17:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1200443392 bytes 2024-04-13T13:17:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1249997824 bytes 2024-04-13T13:17:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1308375040 bytes 2024-04-13T13:17:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1338276864 bytes 2024-04-13T13:17:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1373545472 bytes 2024-04-13T13:17:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1374151680 bytes 2024-04-13T13:17:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1395796992 bytes 2024-04-13T13:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1404977152 bytes 2024-04-13T13:17:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1439614976 bytes 2024-04-13T13:17:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1453905920 bytes 2024-04-13T13:17:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1491330048 bytes 2024-04-13T13:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1527265280 bytes 2024-04-13T13:17:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1544104960 bytes 2024-04-13T13:17:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1590632448 bytes 2024-04-13T13:17:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1613906944 bytes 2024-04-13T13:17:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1695987712 bytes 2024-04-13T13:17:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1714900992 bytes 2024-04-13T13:17:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1793903616 bytes 2024-04-13T13:17:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1808754688 bytes 2024-04-13T13:17:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1832086528 bytes 2024-04-13T13:17:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1868008448 bytes 2024-04-13T13:17:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1907775488 bytes 2024-04-13T13:17:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 1933365248 bytes 2024-04-13T13:17:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 1986497536 bytes 2024-04-13T13:17:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2025379840 bytes 2024-04-13T13:17:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2040915968 bytes 2024-04-13T13:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2043515904 bytes 2024-04-13T13:17:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2125043712 bytes 2024-04-13T13:17:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2172722176 bytes 2024-04-13T13:17:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2245610496 bytes 2024-04-13T13:17:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2277242880 bytes 2024-04-13T13:17:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2329910272 bytes 2024-04-13T13:17:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2367774720 bytes 2024-04-13T13:17:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2428463104 bytes 2024-04-13T13:17:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2497175552 bytes 2024-04-13T13:17:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2520313856 bytes 2024-04-13T13:17:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2547933184 bytes 2024-04-13T13:17:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2548544512 bytes 2024-04-13T13:17:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2626225152 bytes 2024-04-13T13:17:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2686846976 bytes 2024-04-13T13:17:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2716730368 bytes 2024-04-13T13:17:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:17:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:17:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:17:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:17:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535941120.0,"last_snapshot_bytes":535941120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014151,"remote_snapshot_timestamp":1713014151,"replay_state":"idle"} 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:17:17 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:17:17' 2024-04-13T13:17:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:17:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:17:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:17:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535941120.0,"last_snapshot_bytes":535941120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014151,"remote_snapshot_timestamp":1713014151,"replay_state":"idle"} 2024-04-13T13:17:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:17:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:17:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:17:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:17:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:17:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:17:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:17:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:17:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:17:17' 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:17:17' 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:17:17' 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:17:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:17:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42194267mirror.2smithi12218.2.2-1187-gd540ebactrueOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOK43774380mirror.0smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535941120.0,"last_snapshot_bytes":535941120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014151,"remote_snapshot_timestamp":1713014151,"replay_state":"idle"}42144264mirror.1smithi1222024-04-13 13:17:17cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:17:17' 2024-04-13T13:17:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:17:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:17:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:17:17' 2024-04-13T13:17:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:17:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:17:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:17:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:17:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:17:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:17:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap4 2024-04-13T13:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap4 2024-04-13T13:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap4 2024-04-13T13:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:17:59.676 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6591330739218131) > weight (0.25) 2024-04-13T13:17:59.676 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-13T13:17:59.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-13T13:17:59.677 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-13T13:17:59.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi122.stderr:2024-04-13T13:17:59.677+0000 7fd39e5db640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41419) UID: 1000 2024-04-13T13:18:05.677 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-13T13:18:05.677 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:18:05.678 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-13T13:18:05.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-13T13:18:05.678 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-13T13:18:05.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-13T13:18:05.681 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-13T13:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:18:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:18:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:18:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:18:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:18:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:18:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:18:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:18:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:18:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:18:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:18:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:18:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:18:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:18:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:18:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:18:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:18:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:18:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:18:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:18:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:18:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:18:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:18:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:18:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:18:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:18:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:18:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:18:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:18:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:18:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:18:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:18:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:18:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:18:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:18:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:18:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:18:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:18:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:18:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 17 2024-04-13T13:18:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:18:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:18:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:18:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:18:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:18:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:18:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=17 2024-04-13T13:18:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:18:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:18:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:18:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:18:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:18:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:18:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:18:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:18:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:18:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:18:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:18:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 17 = '' 2024-04-13T13:18:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:18:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:18:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:18:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:18:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:18:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:18:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:18:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:18:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:18:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:18:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 17 = '' 2024-04-13T13:18:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:18:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:18:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:18:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:18:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:18:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:18:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:18:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:18:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:18:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:18:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 17 = '' 2024-04-13T13:18:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:18:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:18:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:18:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:18:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:18:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:18:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=17 2024-04-13T13:18:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 17 = 17 2024-04-13T13:18:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:18:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-04-13T13:18:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap4 2024-04-13T13:18:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:18:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:18:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:18:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 8 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:18:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:18:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:16:41 2024 2024-04-13T13:18:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:18:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:18:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:18:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:18:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:18:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:18:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:18:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap3-clean ']' 2024-04-13T13:18:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-13T13:18:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap3-clean 2024-04-13T13:18:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:18:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:18:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:18:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:18:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:18:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:18:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap4 2024-04-13T13:18:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap4 2024-04-13T13:18:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:18:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:18:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:18:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:18:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:18:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:18:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:18:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap4-clean 2024-04-13T13:18:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-04-13T13:18:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:18:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap4-clean 2024-04-13T13:18:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:18:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:18:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:18:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:18:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:18:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=9 2024-04-13T13:18:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:18:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:18:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 50960384 bytes 2024-04-13T13:18:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 111545344 bytes 2024-04-13T13:18:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 143135744 bytes 2024-04-13T13:18:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 204233728 bytes 2024-04-13T13:18:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 204833792 bytes 2024-04-13T13:18:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 217005056 bytes 2024-04-13T13:18:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 238825472 bytes 2024-04-13T13:18:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 296487936 bytes 2024-04-13T13:18:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 332844032 bytes 2024-04-13T13:18:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 357791744 bytes 2024-04-13T13:18:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 398706688 bytes 2024-04-13T13:18:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 411728896 bytes 2024-04-13T13:18:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 512896000 bytes 2024-04-13T13:18:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 579732480 bytes 2024-04-13T13:18:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 619527168 bytes 2024-04-13T13:18:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 658633728 bytes 2024-04-13T13:18:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 703837184 bytes 2024-04-13T13:18:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 704450560 bytes 2024-04-13T13:18:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 768716800 bytes 2024-04-13T13:18:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 798044160 bytes 2024-04-13T13:18:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 843577344 bytes 2024-04-13T13:18:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 858910720 bytes 2024-04-13T13:18:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 904501248 bytes 2024-04-13T13:18:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 927141888 bytes 2024-04-13T13:18:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1006593024 bytes 2024-04-13T13:18:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1075567616 bytes 2024-04-13T13:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1145607168 bytes 2024-04-13T13:18:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1239521280 bytes 2024-04-13T13:18:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1267865600 bytes 2024-04-13T13:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1309822976 bytes 2024-04-13T13:18:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1334792192 bytes 2024-04-13T13:18:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1382396928 bytes 2024-04-13T13:18:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1437394944 bytes 2024-04-13T13:18:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1462686720 bytes 2024-04-13T13:18:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1480348672 bytes 2024-04-13T13:18:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1521381376 bytes 2024-04-13T13:18:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1556422656 bytes 2024-04-13T13:18:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1607760896 bytes 2024-04-13T13:18:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1637708800 bytes 2024-04-13T13:18:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1694840832 bytes 2024-04-13T13:18:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1783869440 bytes 2024-04-13T13:18:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1807996928 bytes 2024-04-13T13:18:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1842055168 bytes 2024-04-13T13:18:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1878924288 bytes 2024-04-13T13:18:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1922072576 bytes 2024-04-13T13:18:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1954974720 bytes 2024-04-13T13:18:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1980179456 bytes 2024-04-13T13:18:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2033616896 bytes 2024-04-13T13:18:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2111951872 bytes 2024-04-13T13:18:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2123542528 bytes 2024-04-13T13:18:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2151131136 bytes 2024-04-13T13:18:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2152053760 bytes 2024-04-13T13:18:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2154132480 bytes 2024-04-13T13:18:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2198554624 bytes 2024-04-13T13:18:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2243671040 bytes 2024-04-13T13:18:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2281164800 bytes 2024-04-13T13:18:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2308237312 bytes 2024-04-13T13:18:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2366726144 bytes 2024-04-13T13:18:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2387124224 bytes 2024-04-13T13:18:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2434821120 bytes 2024-04-13T13:18:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2484426752 bytes 2024-04-13T13:18:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:18:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:18:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:18:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:18:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:18:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:18:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:18:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:18:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:18:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:18:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:18:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:18:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:18:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:18:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:18:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:18:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:18:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:18:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:18:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:18:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531389952.0,"last_snapshot_bytes":526838784,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014245,"remote_snapshot_timestamp":1713014245,"replay_state":"idle"} 2024-04-13T13:18:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2024-04-13T13:18:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:18:17 2024-04-13T13:18:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:18:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:18:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:18:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:18:17' 2024-04-13T13:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:18:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531389952.0,"last_snapshot_bytes":526838784,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014245,"remote_snapshot_timestamp":1713014245,"replay_state":"idle"} 2024-04-13T13:18:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:18:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:18:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:18:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:18:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:18:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:18:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:18:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:18:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:18:17' 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:18:17' 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:18:17' 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:18:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:18:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42194267mirror.2smithi12218.2.2-1187-gd540ebactrueOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOK43774380mirror.0smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531389952.0,"last_snapshot_bytes":526838784,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014245,"remote_snapshot_timestamp":1713014245,"replay_state":"idle"}42144264mirror.1smithi1222024-04-13 13:18:17cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:18:17' 2024-04-13T13:18:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:18:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:18:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:18:17' 2024-04-13T13:18:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:18:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:18:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:18:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:18:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:18:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:18:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:18:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap5 2024-04-13T13:18:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap5 2024-04-13T13:18:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:18:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:18:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:18:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap5 2024-04-13T13:18:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:19:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:19:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:19:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:19:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:19:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:19:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:19:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:19:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:19:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:19:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:19:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:19:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:19:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:19:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:19:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:19:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:19:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:19:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:19:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:19:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:19:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:19:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:19:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:19:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:19:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:19:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:19:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:19:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:19:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:19:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:19:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:19:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:19:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:19:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:19:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:19:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:19:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:19:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:19:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:19:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:19:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:19:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:19:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:19:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:19:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:19:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:19:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:19:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:19:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:19:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:19:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:19:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:19:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:19:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:19:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:19:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 21 2024-04-13T13:19:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:19:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:19:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:19:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:19:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:19:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:19:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=21 2024-04-13T13:19:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:19:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:19:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:19:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:19:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:19:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:19:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:19:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:19:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:19:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:19:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:19:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 21 = '' 2024-04-13T13:19:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:19:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:19:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:19:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:19:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:19:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:19:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:19:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:19:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:19:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:19:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 21 = '' 2024-04-13T13:19:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:19:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:19:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:19:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:19:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:19:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:19:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:19:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:19:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:19:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:19:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 21 = '' 2024-04-13T13:19:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:19:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:19:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:19:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:19:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:19:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:19:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=21 2024-04-13T13:19:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 21 = 21 2024-04-13T13:19:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:19:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-04-13T13:19:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap5 2024-04-13T13:19:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:19:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:19:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:19:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 10 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:18:24 2024 2024-04-13T13:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap4-clean ']' 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap4-clean 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:19:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:19:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:19:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap5 2024-04-13T13:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap5 2024-04-13T13:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:19:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:19:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:19:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap5-clean 2024-04-13T13:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-04-13T13:19:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:19:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap5-clean 2024-04-13T13:19:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:19:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:19:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:19:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:19:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:19:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=11 2024-04-13T13:19:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:19:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:19:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 58425344 bytes 2024-04-13T13:19:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 119383040 bytes 2024-04-13T13:19:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 120006656 bytes 2024-04-13T13:19:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 156027904 bytes 2024-04-13T13:19:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 166796288 bytes 2024-04-13T13:19:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 214888448 bytes 2024-04-13T13:19:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 314966016 bytes 2024-04-13T13:19:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 327544832 bytes 2024-04-13T13:19:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 367938560 bytes 2024-04-13T13:19:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 397803520 bytes 2024-04-13T13:19:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 427360256 bytes 2024-04-13T13:19:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 430067712 bytes 2024-04-13T13:19:20.683 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.921696569886316) > weight (0.25) 2024-04-13T13:19:20.683 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6205503821717337) > weight (0.25) 2024-04-13T13:19:20.683 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-13T13:19:20.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-13T13:19:20.684 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-13T13:19:20.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi122.stderr:2024-04-13T13:19:20.684+0000 7f62c5bf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 39701) UID: 1000 2024-04-13T13:19:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 489052160 bytes 2024-04-13T13:19:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 525991936 bytes 2024-04-13T13:19:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 548303872 bytes 2024-04-13T13:19:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 569565184 bytes 2024-04-13T13:19:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 588591104 bytes 2024-04-13T13:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 612574208 bytes 2024-04-13T13:19:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 655038464 bytes 2024-04-13T13:19:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 674065408 bytes 2024-04-13T13:19:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 732974080 bytes 2024-04-13T13:19:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 751415296 bytes 2024-04-13T13:19:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 767488000 bytes 2024-04-13T13:19:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 799381504 bytes 2024-04-13T13:19:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 835864576 bytes 2024-04-13T13:19:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 880924672 bytes 2024-04-13T13:19:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 933764096 bytes 2024-04-13T13:19:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 996339712 bytes 2024-04-13T13:19:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1048616960 bytes 2024-04-13T13:19:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1075562496 bytes 2024-04-13T13:19:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1097373696 bytes 2024-04-13T13:19:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1144734720 bytes 2024-04-13T13:19:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1157737472 bytes 2024-04-13T13:19:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1204075520 bytes 2024-04-13T13:19:23.684 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-13T13:19:23.685 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:19:23.685 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-13T13:19:23.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-13T13:19:23.685 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-13T13:19:23.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-13T13:19:23.690 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-04-13T13:19:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1211980800 bytes 2024-04-13T13:19:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1290768384 bytes 2024-04-13T13:19:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1326658560 bytes 2024-04-13T13:19:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1384744960 bytes 2024-04-13T13:19:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1430531072 bytes 2024-04-13T13:19:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1467584512 bytes 2024-04-13T13:19:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1487156224 bytes 2024-04-13T13:19:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1567299584 bytes 2024-04-13T13:19:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1651585024 bytes 2024-04-13T13:19:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1674704896 bytes 2024-04-13T13:19:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1743450112 bytes 2024-04-13T13:19:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1797661696 bytes 2024-04-13T13:19:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1824719872 bytes 2024-04-13T13:19:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1840020480 bytes 2024-04-13T13:19:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1845853184 bytes 2024-04-13T13:19:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1877957632 bytes 2024-04-13T13:19:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1888908288 bytes 2024-04-13T13:19:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1950128128 bytes 2024-04-13T13:19:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 1986174976 bytes 2024-04-13T13:19:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2089336832 bytes 2024-04-13T13:19:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2161536000 bytes 2024-04-13T13:19:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2219895808 bytes 2024-04-13T13:19:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2240356352 bytes 2024-04-13T13:19:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2244276224 bytes 2024-04-13T13:19:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2299343872 bytes 2024-04-13T13:19:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2338751488 bytes 2024-04-13T13:19:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2339352576 bytes 2024-04-13T13:19:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2382635008 bytes 2024-04-13T13:19:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2438624256 bytes 2024-04-13T13:19:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2503670784 bytes 2024-04-13T13:19:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2537833472 bytes 2024-04-13T13:19:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2589103104 bytes 2024-04-13T13:19:28.677 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-13T13:19:28.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-13T13:19:28.677 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-13T13:19:28.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi122.stderr:2024-04-13T13:19:28.678+0000 7f22651f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 42115) UID: 1000 2024-04-13T13:19:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2605744128 bytes 2024-04-13T13:19:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2645980160 bytes 2024-04-13T13:19:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2665671680 bytes 2024-04-13T13:19:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2728935424 bytes 2024-04-13T13:19:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2815516672 bytes 2024-04-13T13:19:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2884286464 bytes 2024-04-13T13:19:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2896541696 bytes 2024-04-13T13:19:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:19:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:19:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:19:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:19:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:19:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:19:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":17541188.27,"bytes_per_snapshot":530982912.0,"last_snapshot_bytes":526235648,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014349,"remote_snapshot_timestamp":1713014349,"replay_state":"idle"} 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:19:17 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:19:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:19:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:19:17' 2024-04-13T13:19:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:19:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:19:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:19:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":17541188.27,"bytes_per_snapshot":530982912.0,"last_snapshot_bytes":526235648,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014349,"remote_snapshot_timestamp":1713014349,"replay_state":"idle"} 2024-04-13T13:19:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:19:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:19:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:19:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:19:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:19:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:19:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:19:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:19:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:19:17' 2024-04-13T13:19:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:19:17' 2024-04-13T13:19:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:19:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:19:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:19:17' 2024-04-13T13:19:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:19:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:19:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:19:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42194267mirror.2smithi12218.2.2-1187-gd540ebactrueOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":17541188.27,"bytes_per_snapshot":530982912.0,"last_snapshot_bytes":526235648,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014349,"remote_snapshot_timestamp":1713014349,"replay_state":"idle"}42144264mirror.1smithi1222024-04-13 13:19:17cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:19:17' 2024-04-13T13:19:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:19:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:19:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:19:17' 2024-04-13T13:19:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:19:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:19:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:19:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:19:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:19:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:19:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:19:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap6 2024-04-13T13:19:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap6 2024-04-13T13:19:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:19:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:19:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:19:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap6 2024-04-13T13:19:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:19:32.678 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-13T13:19:32.679 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:19:32.679 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-13T13:19:32.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-13T13:19:32.679 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-13T13:19:32.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-13T13:19:32.682 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-04-13T13:20:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:20:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:20:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:20:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:20:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:20:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:20:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:20:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:20:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:20:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:20:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:20:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:20:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:20:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:20:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:20:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:20:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:20:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:20:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:20:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:20:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:20:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:20:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:20:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:20:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:20:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:20:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:20:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:20:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:20:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:20:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:20:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:20:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:20:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:20:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:20:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:20:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:20:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:20:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:20:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:20:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:20:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:20:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:20:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:20:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:20:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:20:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:20:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:20:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:20:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:20:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:20:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 25 2024-04-13T13:20:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:20:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:20:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:20:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:20:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=25 2024-04-13T13:20:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:20:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:20:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:20:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:20:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:20:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:20:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:20:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:20:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:20:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:20:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 25 = '' 2024-04-13T13:20:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:20:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:20:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:20:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:20:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:20:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:20:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:20:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:20:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:20:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 25 = '' 2024-04-13T13:20:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:20:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:20:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:20:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:20:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:20:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:20:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:20:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:20:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:20:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 25 = '' 2024-04-13T13:20:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:20:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:20:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:20:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:20:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=25 2024-04-13T13:20:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 25 = 25 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap6 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:20:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:20:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-04-13T13:20:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 12 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:20:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:18:24 2024 2024-04-13T13:20:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:20:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:20:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:20:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:20:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:20:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:20:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap5-clean ']' 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap5-clean 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:20:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:20:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:20:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap6 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap6 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:20:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:20:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:20:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:20:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap6-clean 2024-04-13T13:20:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-04-13T13:20:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap6-clean 2024-04-13T13:20:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:20:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:20:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:20:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:20:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:20:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:20:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:20:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 37596160 bytes 2024-04-13T13:20:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 87998464 bytes 2024-04-13T13:20:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 125514752 bytes 2024-04-13T13:20:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 189778944 bytes 2024-04-13T13:20:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 195772416 bytes 2024-04-13T13:20:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 261592064 bytes 2024-04-13T13:20:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 277168128 bytes 2024-04-13T13:20:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 291338240 bytes 2024-04-13T13:20:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 301675520 bytes 2024-04-13T13:20:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 337862656 bytes 2024-04-13T13:20:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 380078080 bytes 2024-04-13T13:20:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 432831488 bytes 2024-04-13T13:20:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 500848640 bytes 2024-04-13T13:20:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 501453824 bytes 2024-04-13T13:20:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 546829312 bytes 2024-04-13T13:20:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 618226688 bytes 2024-04-13T13:20:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 642301952 bytes 2024-04-13T13:20:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 730644480 bytes 2024-04-13T13:20:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 838459392 bytes 2024-04-13T13:20:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 892346368 bytes 2024-04-13T13:20:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 979788800 bytes 2024-04-13T13:20:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1004915712 bytes 2024-04-13T13:20:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1037205504 bytes 2024-04-13T13:20:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1068581888 bytes 2024-04-13T13:20:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1101721600 bytes 2024-04-13T13:20:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1141069824 bytes 2024-04-13T13:20:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1176435712 bytes 2024-04-13T13:20:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1255079936 bytes 2024-04-13T13:20:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1255682048 bytes 2024-04-13T13:20:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1286746112 bytes 2024-04-13T13:20:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1360648192 bytes 2024-04-13T13:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1429565440 bytes 2024-04-13T13:20:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1487473664 bytes 2024-04-13T13:20:16.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1501376512 bytes 2024-04-13T13:20:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1549588480 bytes 2024-04-13T13:20:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1595332608 bytes 2024-04-13T13:20:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1626392576 bytes 2024-04-13T13:20:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1709984768 bytes 2024-04-13T13:20:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1780149248 bytes 2024-04-13T13:20:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1823114240 bytes 2024-04-13T13:20:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1870143488 bytes 2024-04-13T13:20:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1910434816 bytes 2024-04-13T13:20:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1962621952 bytes 2024-04-13T13:20:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 2025191424 bytes 2024-04-13T13:20:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 2081350656 bytes 2024-04-13T13:20:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2212396032 bytes 2024-04-13T13:20:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2286159872 bytes 2024-04-13T13:20:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2363112448 bytes 2024-04-13T13:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2400351232 bytes 2024-04-13T13:20:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2430904320 bytes 2024-04-13T13:20:20.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2467398656 bytes 2024-04-13T13:20:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2533876736 bytes 2024-04-13T13:20:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2559994880 bytes 2024-04-13T13:20:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2567683072 bytes 2024-04-13T13:20:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2568270848 bytes 2024-04-13T13:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2635537408 bytes 2024-04-13T13:20:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2707205120 bytes 2024-04-13T13:20:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2746743808 bytes 2024-04-13T13:20:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2754753536 bytes 2024-04-13T13:20:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2859120640 bytes 2024-04-13T13:20:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2891244544 bytes 2024-04-13T13:20:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2906205184 bytes 2024-04-13T13:20:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2924384256 bytes 2024-04-13T13:20:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 3024301056 bytes 2024-04-13T13:20:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 3125518336 bytes 2024-04-13T13:20:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 3208892416 bytes 2024-04-13T13:20:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 3256817664 bytes 2024-04-13T13:20:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 3300980736 bytes 2024-04-13T13:20:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 3400893440 bytes 2024-04-13T13:20:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 3487702016 bytes 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:20:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:20:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:20:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:20:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:20:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:20:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":17840435.2,"bytes_per_snapshot":530724352.0,"last_snapshot_bytes":535213056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014402,"remote_snapshot_timestamp":1713014402,"replay_state":"idle"} 2024-04-13T13:20:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2024-04-13T13:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:20:17 2024-04-13T13:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:20:17' 2024-04-13T13:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:20:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:20:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":17840435.2,"bytes_per_snapshot":530724352.0,"last_snapshot_bytes":535213056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014402,"remote_snapshot_timestamp":1713014402,"replay_state":"idle"} 2024-04-13T13:20:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:20:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:20:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:20:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:20:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:20:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:20:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:20:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:20:17' 2024-04-13T13:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:20:17' 2024-04-13T13:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:20:17' 2024-04-13T13:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:20:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42194267mirror.2smithi12218.2.2-1187-gd540ebactrueOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOK45214524mirror.0smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":17840435.2,"bytes_per_snapshot":530724352.0,"last_snapshot_bytes":535213056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014402,"remote_snapshot_timestamp":1713014402,"replay_state":"idle"}42144264mirror.1smithi1222024-04-13 13:20:17cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:20:17' 2024-04-13T13:20:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:20:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:20:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:20:17' 2024-04-13T13:20:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:20:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:20:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:20:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:20:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:20:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap7 2024-04-13T13:20:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap7 2024-04-13T13:20:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap7 2024-04-13T13:20:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:20:48.691 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-13T13:20:48.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-13T13:20:48.691 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-13T13:20:48.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr:2024-04-13T13:20:48.694+0000 7f1fff7f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 39668) UID: 1000 2024-04-13T13:20:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:20:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:20:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:20:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:20:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:20:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:20:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:20:53.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:20:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:20:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:20:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:20:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:20:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:20:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:20:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:20:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:20:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:20:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:20:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:20:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:20:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:20:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:20:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:20:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:20:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:20:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:20:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:20:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:20:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:20:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:20:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:20:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:20:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:20:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:20:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:20:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:20:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:20:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:20:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:20:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:20:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:20:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:20:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:20:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:20:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:20:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 29 2024-04-13T13:20:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:20:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:20:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:20:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:20:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:20:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=29 2024-04-13T13:20:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:20:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:20:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:20:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:20:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:20:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:20:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:20:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:20:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:20:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:20:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 29 = '' 2024-04-13T13:20:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:20:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:20:56.692 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-13T13:20:56.692 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:20:56.693 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-13T13:20:56.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-13T13:20:56.693 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-13T13:20:56.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-13T13:20:56.696 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-04-13T13:20:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:20:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:20:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:20:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:20:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:20:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:20:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:20:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 29 = '' 2024-04-13T13:20:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:20:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:20:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:20:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:20:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:20:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:20:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:20:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:20:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:20:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 29 = '' 2024-04-13T13:20:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:20:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:20:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:20:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:20:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:20:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:20:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=29 2024-04-13T13:20:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 29 = 29 2024-04-13T13:20:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:20:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-04-13T13:20:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:20:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:20:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:20:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap7 2024-04-13T13:20:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:20:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:20:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:21:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 14 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:20:08 2024 2024-04-13T13:21:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:21:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:21:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:21:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:21:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:21:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:21:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap6-clean ']' 2024-04-13T13:21:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-13T13:21:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap6-clean 2024-04-13T13:21:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:21:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:21:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:21:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:21:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:21:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap7 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap7 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:21:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:21:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:21:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap7-clean 2024-04-13T13:21:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-04-13T13:21:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap7-clean 2024-04-13T13:21:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:21:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:21:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:21:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:21:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:21:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:21:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:21:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 62660608 bytes 2024-04-13T13:21:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 144344064 bytes 2024-04-13T13:21:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 180648960 bytes 2024-04-13T13:21:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 206031872 bytes 2024-04-13T13:21:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 283794432 bytes 2024-04-13T13:21:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 297878528 bytes 2024-04-13T13:21:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 344175616 bytes 2024-04-13T13:21:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 412843008 bytes 2024-04-13T13:21:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 464117760 bytes 2024-04-13T13:21:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 493902848 bytes 2024-04-13T13:21:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 552481792 bytes 2024-04-13T13:21:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 617169920 bytes 2024-04-13T13:21:06.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 653939712 bytes 2024-04-13T13:21:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 743809024 bytes 2024-04-13T13:21:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 769710080 bytes 2024-04-13T13:21:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 783214592 bytes 2024-04-13T13:21:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 860135424 bytes 2024-04-13T13:21:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 939270144 bytes 2024-04-13T13:21:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 985719808 bytes 2024-04-13T13:21:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 1014316032 bytes 2024-04-13T13:21:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 1112031232 bytes 2024-04-13T13:21:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1193346048 bytes 2024-04-13T13:21:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1250125824 bytes 2024-04-13T13:21:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1312608256 bytes 2024-04-13T13:21:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1346013184 bytes 2024-04-13T13:21:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1346611200 bytes 2024-04-13T13:21:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1395234816 bytes 2024-04-13T13:21:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1441293312 bytes 2024-04-13T13:21:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1477724160 bytes 2024-04-13T13:21:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1574576128 bytes 2024-04-13T13:21:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1588912128 bytes 2024-04-13T13:21:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1635705856 bytes 2024-04-13T13:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1656048640 bytes 2024-04-13T13:21:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1682587648 bytes 2024-04-13T13:21:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1714799616 bytes 2024-04-13T13:21:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1782699008 bytes 2024-04-13T13:21:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1816212480 bytes 2024-04-13T13:21:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1859137536 bytes 2024-04-13T13:21:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1917223936 bytes 2024-04-13T13:21:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1946985472 bytes 2024-04-13T13:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 2075492352 bytes 2024-04-13T13:21:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 2099035136 bytes 2024-04-13T13:21:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 2135841792 bytes 2024-04-13T13:21:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 2137792512 bytes 2024-04-13T13:21:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 2166735872 bytes 2024-04-13T13:21:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2214212608 bytes 2024-04-13T13:21:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2260910080 bytes 2024-04-13T13:21:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2304408576 bytes 2024-04-13T13:21:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2340233216 bytes 2024-04-13T13:21:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2367399936 bytes 2024-04-13T13:21:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2413125632 bytes 2024-04-13T13:21:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2499788800 bytes 2024-04-13T13:21:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2530722816 bytes 2024-04-13T13:21:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2546074624 bytes 2024-04-13T13:21:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2594659328 bytes 2024-04-13T13:21:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2679603200 bytes 2024-04-13T13:21:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2692557824 bytes 2024-04-13T13:21:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2706662400 bytes 2024-04-13T13:21:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2730962944 bytes 2024-04-13T13:21:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2732033024 bytes 2024-04-13T13:21:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2732634112 bytes 2024-04-13T13:21:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2750876672 bytes 2024-04-13T13:21:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2771764224 bytes 2024-04-13T13:21:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2787739648 bytes 2024-04-13T13:21:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2807932928 bytes 2024-04-13T13:21:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2841083904 bytes 2024-04-13T13:21:14.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2915211264 bytes 2024-04-13T13:21:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2954991616 bytes 2024-04-13T13:21:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 3073486848 bytes 2024-04-13T13:21:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 3110113280 bytes 2024-04-13T13:21:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 3119403008 bytes 2024-04-13T13:21:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 3160363008 bytes 2024-04-13T13:21:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 3209292800 bytes 2024-04-13T13:21:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 3214340096 bytes 2024-04-13T13:21:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 3216847872 bytes 2024-04-13T13:21:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3237820416 bytes 2024-04-13T13:21:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:21:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:21:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:21:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:21:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:21:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:21:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:21:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:21:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:21:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:21:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:21:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:21:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:21:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:21:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530724352.0,"last_snapshot_bytes":535213056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014402,"remote_snapshot_timestamp":1713014402,"replay_state":"idle"} 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:20:47 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: ' 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:21:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:21:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:21:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530724352.0,"last_snapshot_bytes":535213056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014402,"remote_snapshot_timestamp":1713014402,"replay_state":"idle"} 2024-04-13T13:21:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:21:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:21:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:21:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:21:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:21:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:21:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:21:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:21:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:20:47' 2024-04-13T13:21:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:20:47' 2024-04-13T13:21:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:21:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:20:47' 2024-04-13T13:21:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:21:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:21:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:21:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='WARNINGOKWARNING142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42194267mirror.2smithi12218.2.2-1187-gd540ebactrueOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOK45214524mirror.0smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530724352.0,"last_snapshot_bytes":535213056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014402,"remote_snapshot_timestamp":1713014402,"replay_state":"idle"}42144264mirror.1smithi1222024-04-13 13:20:47cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57edown+unknownstatus not found' 2024-04-13T13:21:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:21:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:21:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:20:47' 2024-04-13T13:21:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:21:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:21:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:21:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:21:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap8 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap8 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap8 2024-04-13T13:21:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:21:24.684 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-13T13:21:24.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-13T13:21:24.684 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-13T13:21:24.684 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-13T13:21:24.684 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:21:24.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi122.stderr:2024-04-13T13:21:24.687+0000 7f480edf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 44238) UID: 1000 2024-04-13T13:21:30.787 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-13T13:21:30.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-13T13:21:30.787 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-13T13:21:30.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-13T13:21:30.791 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-04-13T13:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:21:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:21:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:21:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:21:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:21:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:21:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:21:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:21:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:21:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:21:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:21:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:21:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:21:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:21:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:21:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:21:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:21:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:21:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:21:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:21:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:21:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:21:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:21:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:21:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:21:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:21:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:21:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:21:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:21:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:21:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:21:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:21:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:21:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:21:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:21:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:21:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:21:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:21:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:21:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:21:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:21:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:21:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:21:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:21:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:21:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:21:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:21:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:21:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:21:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:21:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:21:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:21:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:21:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:21:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:21:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:21:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 33 2024-04-13T13:21:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:21:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:21:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:21:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:21:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:21:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=33 2024-04-13T13:21:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:21:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:21:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:21:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:21:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:21:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:21:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:21:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:21:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:21:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:21:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:21:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 33 = '' 2024-04-13T13:21:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:21:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:21:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:21:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:21:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:21:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:21:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:21:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:21:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:21:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:21:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 33 = '' 2024-04-13T13:21:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:21:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:21:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:21:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:21:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:21:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:21:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:21:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:21:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:21:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:21:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 33 = '' 2024-04-13T13:21:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:21:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:21:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:21:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:21:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:21:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:21:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=33 2024-04-13T13:21:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 33 = 33 2024-04-13T13:21:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-04-13T13:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap8 2024-04-13T13:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:21:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-04-13T13:21:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 16 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:20:08 2024 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:21:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:21:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:21:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:21:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:21:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap7-clean ']' 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap7-clean 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:21:54.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:21:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:21:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap8 2024-04-13T13:21:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap8 2024-04-13T13:21:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:21:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:21:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:21:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:21:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:21:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:21:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:21:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap8-clean 2024-04-13T13:21:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-04-13T13:21:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap8-clean 2024-04-13T13:21:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:21:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:21:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:21:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:21:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:21:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:21:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:21:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=13 2024-04-13T13:21:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:21:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:21:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 31410176 bytes 2024-04-13T13:21:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 73304064 bytes 2024-04-13T13:21:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 149231616 bytes 2024-04-13T13:21:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 184299520 bytes 2024-04-13T13:21:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 205670400 bytes 2024-04-13T13:21:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 240569344 bytes 2024-04-13T13:21:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 288486400 bytes 2024-04-13T13:21:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 354285568 bytes 2024-04-13T13:21:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 373854208 bytes 2024-04-13T13:21:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 411364352 bytes 2024-04-13T13:21:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 444030976 bytes 2024-04-13T13:21:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 466798592 bytes 2024-04-13T13:21:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 504024064 bytes 2024-04-13T13:21:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 558337024 bytes 2024-04-13T13:21:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 574543872 bytes 2024-04-13T13:21:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 611589120 bytes 2024-04-13T13:22:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 618177536 bytes 2024-04-13T13:22:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 628209664 bytes 2024-04-13T13:22:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 685729792 bytes 2024-04-13T13:22:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 693989376 bytes 2024-04-13T13:22:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 703932416 bytes 2024-04-13T13:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 704532480 bytes 2024-04-13T13:22:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 743691264 bytes 2024-04-13T13:22:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 752091136 bytes 2024-04-13T13:22:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 815002624 bytes 2024-04-13T13:22:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 852671488 bytes 2024-04-13T13:22:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 909756416 bytes 2024-04-13T13:22:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 972171264 bytes 2024-04-13T13:22:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1045074944 bytes 2024-04-13T13:22:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1067612160 bytes 2024-04-13T13:22:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1130722304 bytes 2024-04-13T13:22:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1152676864 bytes 2024-04-13T13:22:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1185747968 bytes 2024-04-13T13:22:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1198931968 bytes 2024-04-13T13:22:02.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1236028416 bytes 2024-04-13T13:22:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1296808960 bytes 2024-04-13T13:22:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1390550016 bytes 2024-04-13T13:22:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1438789632 bytes 2024-04-13T13:22:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1488909312 bytes 2024-04-13T13:22:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1495071744 bytes 2024-04-13T13:22:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1518422016 bytes 2024-04-13T13:22:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1534694400 bytes 2024-04-13T13:22:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1660309504 bytes 2024-04-13T13:22:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1698126848 bytes 2024-04-13T13:22:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1720570880 bytes 2024-04-13T13:22:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1755200512 bytes 2024-04-13T13:22:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1761329152 bytes 2024-04-13T13:22:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1802135552 bytes 2024-04-13T13:22:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1865838592 bytes 2024-04-13T13:22:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1892096000 bytes 2024-04-13T13:22:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1900674048 bytes 2024-04-13T13:22:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1941659648 bytes 2024-04-13T13:22:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2004123648 bytes 2024-04-13T13:22:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2087139328 bytes 2024-04-13T13:22:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2102732800 bytes 2024-04-13T13:22:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2150880256 bytes 2024-04-13T13:22:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2210732032 bytes 2024-04-13T13:22:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2259705856 bytes 2024-04-13T13:22:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2303292416 bytes 2024-04-13T13:22:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2359297024 bytes 2024-04-13T13:22:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2410497024 bytes 2024-04-13T13:22:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2437308416 bytes 2024-04-13T13:22:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2504332288 bytes 2024-04-13T13:22:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2597366784 bytes 2024-04-13T13:22:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2627554304 bytes 2024-04-13T13:22:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2662999040 bytes 2024-04-13T13:22:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2701042688 bytes 2024-04-13T13:22:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2812139520 bytes 2024-04-13T13:22:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2850760704 bytes 2024-04-13T13:22:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2907560960 bytes 2024-04-13T13:22:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2956533760 bytes 2024-04-13T13:22:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 3001663488 bytes 2024-04-13T13:22:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 3035283456 bytes 2024-04-13T13:22:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 3057410048 bytes 2024-04-13T13:22:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 3131557888 bytes 2024-04-13T13:22:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3169133568 bytes 2024-04-13T13:22:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 3239097344 bytes 2024-04-13T13:22:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7800 IOs, 3293200384 bytes 2024-04-13T13:22:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7900 IOs, 3325342720 bytes 2024-04-13T13:22:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8000 IOs, 3368280064 bytes 2024-04-13T13:22:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8100 IOs, 3464359936 bytes 2024-04-13T13:22:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:22:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:22:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:22:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:22:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:22:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:22:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:22:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:22:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:22:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:22:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:22:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:22:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:22:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:22:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:22:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:22:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:22:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:22:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:22:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:22:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:22:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:22:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:22:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:22:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534434304.0,"last_snapshot_bytes":533655552,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014454,"remote_snapshot_timestamp":1713014454,"replay_state":"idle"} 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:21:47 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:22:03' 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:22:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:22:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:22:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534434304.0,"last_snapshot_bytes":533655552,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014454,"remote_snapshot_timestamp":1713014454,"replay_state":"idle"} 2024-04-13T13:22:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:22:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:22:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:22:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:22:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:22:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:22:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:22:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:21:47' 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:21:47' 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:21:47' 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:22:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:22:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142144264mirror.1smithi12218.2.2-1187-gd540ebacfalseOK42194267mirror.2smithi12218.2.2-1187-gd540ebactrueOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOK45904593mirror.0smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534434304.0,"last_snapshot_bytes":533655552,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014454,"remote_snapshot_timestamp":1713014454,"replay_state":"idle"}42144264mirror.1smithi1222024-04-13 13:21:47cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:22:03' 2024-04-13T13:22:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:22:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:22:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:21:47' 2024-04-13T13:22:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:22:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:22:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:22:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:22:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:22:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:22:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:22:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap9 2024-04-13T13:22:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap9 2024-04-13T13:22:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:22:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:22:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:22:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap9 2024-04-13T13:22:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:22:33.792 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6353113220399138) > weight (0.25) 2024-04-13T13:22:33.792 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-13T13:22:33.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-13T13:22:33.793 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-13T13:22:33.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:2024-04-13T13:22:33.796+0000 7efdea3db640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 39616) UID: 1000 2024-04-13T13:22:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:22:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:22:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:22:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:22:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:22:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:22:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:22:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:22:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:22:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:22:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:22:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:22:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:22:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:22:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:22:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:22:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:22:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:22:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:22:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:22:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:22:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:22:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:22:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:22:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:22:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:22:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:22:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:22:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:22:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:22:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-13T13:22:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:22:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:22:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:22:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:22:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-13T13:22:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:22:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:22:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:22:40.797 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-13T13:22:40.797 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:22:40.797 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-13T13:22:40.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-13T13:22:40.797 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-13T13:22:40.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-13T13:22:40.800 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-04-13T13:22:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-13T13:22:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:22:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 1 2024-04-13T13:22:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:22:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:22:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:22:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:22:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:22:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:22:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:22:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:22:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:22:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:22:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:22:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:22:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:22:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:22:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:22:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-13T13:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:22:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-13T13:22:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:22:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 2 2024-04-13T13:22:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:22:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:22:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:22:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:22:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:22:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:22:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:22:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:22:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:22:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:22:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:22:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:22:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:22:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:22:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:22:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:22:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:22:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:22:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:22:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:22:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-13T13:22:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:22:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 4 2024-04-13T13:22:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:22:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:22:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:22:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:22:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:22:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:22:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:22:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:22:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:22:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:22:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:22:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:22:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:22:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:22:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:22:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:22:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:22:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-13T13:22:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:22:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:22:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:22:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:22:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:22:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:22:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:22:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:22:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:22:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:22:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:22:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:22:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-13T13:22:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:22:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 8 2024-04-13T13:22:50.697 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.31517271965818505) > weight (0.25) 2024-04-13T13:22:50.697 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2744182514796405) > weight (0.25) 2024-04-13T13:22:50.697 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3925954083398908) > weight (0.25) 2024-04-13T13:22:50.698 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-13T13:22:50.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-13T13:22:50.698 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-13T13:22:50.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi122.stderr:2024-04-13T13:22:50.702+0000 7fd014df9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 42373) UID: 1000 2024-04-13T13:22:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:22:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:22:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:22:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:22:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:22:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:22:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:22:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:22:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:22:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:22:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:22:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:22:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:22:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:22:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:22:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:22:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:22:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:22:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:22:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:22:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-13T13:22:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:22:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:22:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:22:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:22:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-13T13:22:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:22:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:22:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:22:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:22:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 8 2024-04-13T13:22:59.699 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-13T13:22:59.699 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:22:59.699 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-13T13:22:59.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-13T13:22:59.700 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-13T13:22:59.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-13T13:22:59.703 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-04-13T13:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:23:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:23:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:23:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:23:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:23:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:23:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:23:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:23:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:23:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:23:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:23:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:23:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:23:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:23:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:23:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:23:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:23:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:23:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:23:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:23:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:23:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:23:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:23:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:23:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:23:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:23:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:23:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:23:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:23:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:23:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:23:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:23:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:23:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:23:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:23:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:23:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:23:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:23:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:23:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:23:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:23:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:23:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:23:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:23:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:23:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 37 2024-04-13T13:23:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:23:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:23:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:23:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:23:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:23:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:23:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=37 2024-04-13T13:23:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:23:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:23:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:23:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:23:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:23:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:23:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:23:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:23:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:23:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:23:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:23:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 37 = '' 2024-04-13T13:23:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:23:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:23:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:23:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:23:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:23:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:23:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:23:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:23:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:23:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:23:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 37 = '' 2024-04-13T13:23:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:23:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:23:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:23:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:23:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:23:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:23:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:23:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:23:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:23:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:23:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 37 = '' 2024-04-13T13:23:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:23:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:23:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:23:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:23:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:23:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=37 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 37 = 37 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap9 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:23:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:23:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-04-13T13:23:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:23:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:23:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:23:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 18 2024-04-13T13:23:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:21:54 2024 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:23:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap8-clean ']' 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap8-clean 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:23:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:23:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:23:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:23:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap9 2024-04-13T13:23:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap9 2024-04-13T13:23:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:23:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:23:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:23:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:23:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:23:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:23:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap9-clean 2024-04-13T13:23:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-04-13T13:23:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:23:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap9-clean 2024-04-13T13:23:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:23:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:23:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror test 2024-04-13T13:23:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:23:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:23:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:23:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:23:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-13T13:23:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 26869760 bytes 2024-04-13T13:23:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 64985088 bytes 2024-04-13T13:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 153456640 bytes 2024-04-13T13:23:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 195367936 bytes 2024-04-13T13:23:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 257029120 bytes 2024-04-13T13:23:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 276261888 bytes 2024-04-13T13:23:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 323230720 bytes 2024-04-13T13:23:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 383739904 bytes 2024-04-13T13:23:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 474695680 bytes 2024-04-13T13:23:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 501107712 bytes 2024-04-13T13:23:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 523390976 bytes 2024-04-13T13:23:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 534090752 bytes 2024-04-13T13:23:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 607141888 bytes 2024-04-13T13:23:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 621238272 bytes 2024-04-13T13:23:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 634421248 bytes 2024-04-13T13:23:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 691950592 bytes 2024-04-13T13:23:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 709939200 bytes 2024-04-13T13:23:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 768762880 bytes 2024-04-13T13:23:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 769383424 bytes 2024-04-13T13:23:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 868892672 bytes 2024-04-13T13:23:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 890652672 bytes 2024-04-13T13:23:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 937029632 bytes 2024-04-13T13:23:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 959895552 bytes 2024-04-13T13:23:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 985915392 bytes 2024-04-13T13:23:20.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1069794304 bytes 2024-04-13T13:23:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1135594496 bytes 2024-04-13T13:23:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1167210496 bytes 2024-04-13T13:23:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1174110208 bytes 2024-04-13T13:23:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1200057344 bytes 2024-04-13T13:23:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1259913216 bytes 2024-04-13T13:23:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1271094272 bytes 2024-04-13T13:23:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1331759104 bytes 2024-04-13T13:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1332379648 bytes 2024-04-13T13:23:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1369375744 bytes 2024-04-13T13:23:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1434755072 bytes 2024-04-13T13:23:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1471559680 bytes 2024-04-13T13:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1489675264 bytes 2024-04-13T13:23:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1499675648 bytes 2024-04-13T13:23:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1539239936 bytes 2024-04-13T13:23:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1615757312 bytes 2024-04-13T13:23:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1695535104 bytes 2024-04-13T13:23:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1715224576 bytes 2024-04-13T13:23:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1748730880 bytes 2024-04-13T13:23:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1794326528 bytes 2024-04-13T13:23:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1832641536 bytes 2024-04-13T13:23:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1845709824 bytes 2024-04-13T13:23:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1853293568 bytes 2024-04-13T13:23:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1890254848 bytes 2024-04-13T13:23:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1936498688 bytes 2024-04-13T13:23:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1958938624 bytes 2024-04-13T13:23:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1992293376 bytes 2024-04-13T13:23:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2039537664 bytes 2024-04-13T13:23:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2103916544 bytes 2024-04-13T13:23:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2163113984 bytes 2024-04-13T13:23:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2172075008 bytes 2024-04-13T13:23:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2231032832 bytes 2024-04-13T13:23:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2266914816 bytes 2024-04-13T13:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2349722624 bytes 2024-04-13T13:23:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2432616448 bytes 2024-04-13T13:23:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2477816832 bytes 2024-04-13T13:23:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2478420992 bytes 2024-04-13T13:23:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2542871552 bytes 2024-04-13T13:23:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2610093056 bytes 2024-04-13T13:23:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2697550848 bytes 2024-04-13T13:23:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2711104512 bytes 2024-04-13T13:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2725561344 bytes 2024-04-13T13:23:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2769616896 bytes 2024-04-13T13:23:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2800427008 bytes 2024-04-13T13:23:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2853550080 bytes 2024-04-13T13:23:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2868492288 bytes 2024-04-13T13:23:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2905143296 bytes 2024-04-13T13:23:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2988261376 bytes 2024-04-13T13:23:27.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 3019904000 bytes 2024-04-13T13:23:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 3035457536 bytes 2024-04-13T13:23:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 3053153280 bytes 2024-04-13T13:23:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3067762688 bytes 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:23:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:23:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-13T13:23:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='test: 2024-04-13T13:23:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:23:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":40807187.69,"bytes_per_snapshot":530493440.0,"last_snapshot_bytes":530493440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014586,"remote_snapshot_timestamp":1713014586,"replay_state":"idle"} 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:23:17 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2024-04-13 13:23:21' 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ state: .*up+replaying' 2024-04-13T13:23:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:23:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^ description: .*' 2024-04-13T13:23:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":40807187.69,"bytes_per_snapshot":530493440.0,"last_snapshot_bytes":530493440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014586,"remote_snapshot_timestamp":1713014586,"replay_state":"idle"} 2024-04-13T13:23:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:23:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:23:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep '^up+' 2024-04-13T13:23:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:23:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep 'service: *mirror..* on ' 2024-04-13T13:23:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:23:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update 2024-04-13T13:23:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-13T13:23:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:23:17' 2024-04-13T13:23:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-13 13:23:17' 2024-04-13T13:23:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2024-04-13T13:23:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local 'prev_last_update=2024-04-13 13:23:17' 2024-04-13T13:23:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status 2024-04-13T13:23:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ CEPH_ARGS= 2024-04-13T13:23:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-13T13:23:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status='OKOKOK142194267mirror.2smithi12218.2.2-1187-gd540ebactrueOK43174320mirror.3smithi12218.2.2-1187-gd540ebacfalseOK45904593mirror.0smithi12218.2.2-1187-gd540ebacfalseOK46264629mirror.1smithi12218.2.2-1187-gd540ebacfalseOKtest68225865-3024-4487-8d1a-9dd8622491f4up+replayingreplaying, {"bytes_per_second":40807187.69,"bytes_per_snapshot":530493440.0,"last_snapshot_bytes":530493440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1713014586,"remote_snapshot_timestamp":1713014586,"replay_state":"idle"}42194267mirror.2smithi1222024-04-13 13:23:17cluster2702f25c1-769a-41ec-a58d-5bd2b89bc57eup+stoppedlocal image is primary2024-04-13 13:23:21' 2024-04-13T13:23:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2024-04-13T13:23:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-13T13:23:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update='2024-04-13 13:23:17' 2024-04-13T13:23:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-13T13:23:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2024-04-13T13:23:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2024-04-13T13:23:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2024-04-13T13:23:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap10 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap10 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap10 2024-04-13T13:23:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:23:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:24:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:24:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:24:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:24:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:24:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:24:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:24:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:24:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:24:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:24:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:24:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:24:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:24:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:24:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:24:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:24:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:24:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:24:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:24:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:24:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:24:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:24:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:24:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:24:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:24:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:24:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:24:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:24:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:24:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:24:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:24:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:24:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:24:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:24:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:24:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:24:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:24:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:24:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:24:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:24:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:24:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:24:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:24:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:24:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:24:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:24:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:24:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:24:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:24:01.801 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-13T13:24:01.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-13T13:24:01.802 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-13T13:24:01.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi122.stderr:2024-04-13T13:24:01.806+0000 7fb93b3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 45275) UID: 1000 2024-04-13T13:24:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 41 2024-04-13T13:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:24:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:24:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:24:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=41 2024-04-13T13:24:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:24:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:24:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:24:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:24:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:24:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:24:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:24:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 41 = '' 2024-04-13T13:24:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:24:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:24:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:24:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:24:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:24:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:24:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:24:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 41 = '' 2024-04-13T13:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:24:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:24:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:24:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:24:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:24:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:24:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:24:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 41 = '' 2024-04-13T13:24:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:24:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:05.803 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-13T13:24:05.803 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:24:05.803 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-13T13:24:05.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-13T13:24:05.804 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-13T13:24:05.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-13T13:24:05.806 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-04-13T13:24:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:24:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=41 2024-04-13T13:24:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 41 = 41 2024-04-13T13:24:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-04-13T13:24:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap10 2024-04-13T13:24:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:24:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:24:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:24:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-04-13T13:24:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:24:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:24:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 20 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:23:12 2024 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:24:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:24:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:24:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snap9-clean ']' 2024-04-13T13:24:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-13T13:24:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap9-clean 2024-04-13T13:24:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap10 2024-04-13T13:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap10 2024-04-13T13:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clean_snap_name=snap10-clean 2024-04-13T13:24:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-04-13T13:24:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap10-clean 2024-04-13T13:24:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-04-13T13:24:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-13T13:24:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:24:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:24:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-13T13:24:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:24:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:24:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:24:11.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:24:11.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:24:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:24:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:24:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:24:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:24:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:24:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:24:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:24:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:24:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:24:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:24:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:24:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:24:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-13T13:24:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:24:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:24:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:24:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:24:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-13T13:24:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:24:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:24:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:24:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-13T13:24:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-13T13:24:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:24:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:24:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:24:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-13T13:24:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:24:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:24:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:24:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:24:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:24:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:24:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:24:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:24:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:24:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:24:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:24:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-13T13:24:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:24:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:24:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:24:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:24:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:24:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:24:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:24:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-13T13:24:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:24:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-test.status 2024-04-13T13:24:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-test.status 2024-04-13T13:24:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:24:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:24:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-test.status 2024-04-13T13:24:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-test.status 2024-04-13T13:24:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:24:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-13T13:24:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-13T13:24:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 44 2024-04-13T13:24:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:24:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.status 2024-04-13T13:24:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-13T13:24:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:24:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=44 2024-04-13T13:24:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:24:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:24:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:24:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:24:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 44 = '' 2024-04-13T13:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:24:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:24:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:24:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:24:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:24:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:24:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:24:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:24:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 44 = '' 2024-04-13T13:24:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:24:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:24:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.status 2024-04-13T13:24:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-13T13:24:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:24:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=44 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 44 = 44 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap10-clean 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:24:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-04-13T13:24:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2024-04-13T13:24:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 512 MiB in 128 objects 2024-04-13T13:24:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:24:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 21 2024-04-13T13:24:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10a7825cbbfe 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10a7825cbbfe 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:23:12 2024 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:14:19 2024 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 68225865-3024-4487-8d1a-9dd8622491f4 2024-04-13T13:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 10 2024-04-13T13:24:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap1 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap1 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap1 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap1-clean 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap1-clean 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap2 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap2 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap2-clean 2024-04-13T13:24:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-13T13:24:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2-clean 2024-04-13T13:24:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap3 2024-04-13T13:24:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap3 2024-04-13T13:24:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap3 2024-04-13T13:24:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap3-clean 2024-04-13T13:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-13T13:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap3-clean 2024-04-13T13:24:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap4 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap4 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap4 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap4-clean 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap4-clean 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap5 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap5 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap5 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap5-clean 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap5-clean 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap6 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap6 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap6 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap6-clean 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap6-clean 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap7 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap7 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap7 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap7-clean 2024-04-13T13:24:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-13T13:24:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap7-clean 2024-04-13T13:24:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap8 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap8 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap8 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap8-clean 2024-04-13T13:24:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-13T13:24:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap8-clean 2024-04-13T13:24:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap9 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap9 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap9 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap9-clean 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap9-clean 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap10 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap10 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap10 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap10-clean 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror test snap10-clean 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap10-clean 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export 2024-04-13T13:24:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-13T13:24:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-test.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-test.export 2024-04-13T13:24:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:24:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 10 2024-04-13T13:24:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap1 2024-04-13T13:24:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-04-13T13:24:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2024-04-13T13:24:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-04-13T13:24:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap1-clean 2024-04-13T13:24:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-04-13T13:24:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1-clean 2024-04-13T13:24:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-04-13T13:24:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap2 2024-04-13T13:24:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-04-13T13:24:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2024-04-13T13:24:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-04-13T13:24:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap2-clean 2024-04-13T13:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-04-13T13:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2-clean 2024-04-13T13:24:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-04-13T13:24:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap3 2024-04-13T13:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-04-13T13:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap3 2024-04-13T13:24:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-04-13T13:24:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap3-clean 2024-04-13T13:24:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-04-13T13:24:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap3-clean 2024-04-13T13:24:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-04-13T13:24:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap4 2024-04-13T13:24:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-04-13T13:24:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap4 2024-04-13T13:24:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-04-13T13:24:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap4-clean 2024-04-13T13:24:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-04-13T13:24:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap4-clean 2024-04-13T13:24:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-04-13T13:24:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:24:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap5 2024-04-13T13:24:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-04-13T13:24:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap5 2024-04-13T13:24:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-04-13T13:24:58.704 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-13T13:24:58.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-13T13:24:58.705 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-13T13:24:58.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr:2024-04-13T13:24:58.709+0000 7fc0525f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 44820) UID: 1000 2024-04-13T13:24:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap5-clean 2024-04-13T13:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-04-13T13:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap5-clean 2024-04-13T13:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-04-13T13:25:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:25:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap6 2024-04-13T13:25:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-04-13T13:25:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap6 2024-04-13T13:25:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-04-13T13:25:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap6-clean 2024-04-13T13:25:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-04-13T13:25:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap6-clean 2024-04-13T13:25:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-04-13T13:25:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:25:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap7 2024-04-13T13:25:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-04-13T13:25:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap7 2024-04-13T13:25:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-04-13T13:25:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap7-clean 2024-04-13T13:25:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-04-13T13:25:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap7-clean 2024-04-13T13:25:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-04-13T13:25:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:25:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap8 2024-04-13T13:25:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-04-13T13:25:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap8 2024-04-13T13:25:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-04-13T13:25:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap8-clean 2024-04-13T13:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-04-13T13:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap8-clean 2024-04-13T13:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-04-13T13:25:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap9 2024-04-13T13:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-04-13T13:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap9 2024-04-13T13:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-04-13T13:25:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap9-clean 2024-04-13T13:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-04-13T13:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap9-clean 2024-04-13T13:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-04-13T13:25:07.705 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-13T13:25:07.705 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:25:07.705 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-13T13:25:07.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-13T13:25:07.706 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-13T13:25:07.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-13T13:25:07.709 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-04-13T13:25:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 10` 2024-04-13T13:25:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap10 2024-04-13T13:25:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-04-13T13:25:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap10 2024-04-13T13:25:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-04-13T13:25:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap10-clean 2024-04-13T13:25:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-04-13T13:25:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap10-clean 2024-04-13T13:25:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-04-13T13:25:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2024-04-13T13:25:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image_retry cluster2 mirror test 2024-04-13T13:25:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-13T13:25:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2024-04-13T13:25:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image cluster2 mirror test 2024-04-13T13:25:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-04-13T13:25:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-04-13T13:25:10.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi122.stderr:2024-04-13T13:25:10.140+0000 7f46817eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5568df0db040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-13T13:25:10.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi122.stderr:2024-04-13T13:25:10.140+0000 7f4687ff8640 -1 rbd::mirror::ImageReplayer: 0x5568de540280 [3/68225865-3024-4487-8d1a-9dd8622491f4] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-13T13:25:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-13T13:25:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:25:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-04-13T13:25:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:25:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2024-04-13T13:25:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2024-04-13T13:25:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:25:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n '' 2024-04-13T13:25:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ get_image_id cluster1 mirror test 2024-04-13T13:25:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:25:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local pool=mirror 2024-04-13T13:25:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local image=test 2024-04-13T13:25:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-13T13:25:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-13T13:25:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-13T13:25:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2024-04-13T13:25:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-13T13:25:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2024-04-13T13:25:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ get_image_id cluster1 mirror test 2024-04-13T13:25:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:25:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local pool=mirror 2024-04-13T13:25:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local image=test 2024-04-13T13:25:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-13T13:25:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-13T13:25:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-13T13:25:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2024-04-13T13:25:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n '' 2024-04-13T13:25:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2024-04-13T13:25:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:25:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog 'TEST: create many images' 2024-04-13T13:25:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.6pyizPAJcx/rbd-mirror.test.log 2024-04-13T13:25:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ date '+%F %T' 2024-04-13T13:25:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13 13:25:11 TEST: create many images 2024-04-13T13:25:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13 13:25:11 TEST: create many images 2024-04-13T13:25:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap 2024-04-13T13:25:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 50 2024-04-13T13:25:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_1 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_1 128M 2024-04-13T13:25:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2024-04-13T13:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-04-13T13:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-04-13T13:25:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:25:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-04-13T13:25:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_1': 2024-04-13T13:25:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:25:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:25:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:25:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 127b5f2e39a5 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.127b5f2e39a5 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:11 2024 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:11 2024 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:11 2024 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f 2024-04-13T13:25:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:25:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:25:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-13T13:25:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:25:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:25:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=10 2024-04-13T13:25:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:25:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:25:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 22469632 bytes 2024-04-13T13:25:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 120376320 bytes 2024-04-13T13:25:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 146506752 bytes 2024-04-13T13:25:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 228609024 bytes 2024-04-13T13:25:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 297083904 bytes 2024-04-13T13:25:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 327018496 bytes 2024-04-13T13:25:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 375492608 bytes 2024-04-13T13:25:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 392284160 bytes 2024-04-13T13:25:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 428829696 bytes 2024-04-13T13:25:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 433178624 bytes 2024-04-13T13:25:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 505307136 bytes 2024-04-13T13:25:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 533453824 bytes 2024-04-13T13:25:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 534921216 bytes 2024-04-13T13:25:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 577090560 bytes 2024-04-13T13:25:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 624216064 bytes 2024-04-13T13:25:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 650243072 bytes 2024-04-13T13:25:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 658862080 bytes 2024-04-13T13:25:14.807 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4898950759577768) > weight (0.25) 2024-04-13T13:25:14.807 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-13T13:25:14.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-13T13:25:14.808 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-13T13:25:14.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:2024-04-13T13:25:14.813+0000 7f9b97bf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 45796) UID: 1000 2024-04-13T13:25:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 693825536 bytes 2024-04-13T13:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 838162432 bytes 2024-04-13T13:25:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 908724224 bytes 2024-04-13T13:25:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 1000133632 bytes 2024-04-13T13:25:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1052017664 bytes 2024-04-13T13:25:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1167752192 bytes 2024-04-13T13:25:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1219269632 bytes 2024-04-13T13:25:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1219879936 bytes 2024-04-13T13:25:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1220515840 bytes 2024-04-13T13:25:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1249394688 bytes 2024-04-13T13:25:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1277305856 bytes 2024-04-13T13:25:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1280305152 bytes 2024-04-13T13:25:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1285849088 bytes 2024-04-13T13:25:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1353506816 bytes 2024-04-13T13:25:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1376966656 bytes 2024-04-13T13:25:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1449824256 bytes 2024-04-13T13:25:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1504923648 bytes 2024-04-13T13:25:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1563467776 bytes 2024-04-13T13:25:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1568301056 bytes 2024-04-13T13:25:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1592227840 bytes 2024-04-13T13:25:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1625212928 bytes 2024-04-13T13:25:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1659209728 bytes 2024-04-13T13:25:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1683105792 bytes 2024-04-13T13:25:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1688147968 bytes 2024-04-13T13:25:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1696491520 bytes 2024-04-13T13:25:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1762592768 bytes 2024-04-13T13:25:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1789621248 bytes 2024-04-13T13:25:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1810921472 bytes 2024-04-13T13:25:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1811535872 bytes 2024-04-13T13:25:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1872357376 bytes 2024-04-13T13:25:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1897236480 bytes 2024-04-13T13:25:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1959294976 bytes 2024-04-13T13:25:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2021293056 bytes 2024-04-13T13:25:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2095630336 bytes 2024-04-13T13:25:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2173264896 bytes 2024-04-13T13:25:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2206494720 bytes 2024-04-13T13:25:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2219875328 bytes 2024-04-13T13:25:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2270402560 bytes 2024-04-13T13:25:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2324421632 bytes 2024-04-13T13:25:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2359768064 bytes 2024-04-13T13:25:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2422450176 bytes 2024-04-13T13:25:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2507288576 bytes 2024-04-13T13:25:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2539447296 bytes 2024-04-13T13:25:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2607904768 bytes 2024-04-13T13:25:21.809 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-13T13:25:21.809 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:25:21.809 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-13T13:25:21.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-13T13:25:21.809 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-13T13:25:21.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-13T13:25:21.812 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-04-13T13:25:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2632861696 bytes 2024-04-13T13:25:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2651410432 bytes 2024-04-13T13:25:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2656603136 bytes 2024-04-13T13:25:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:25:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:25:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:25:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:25:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:25:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_2 2024-04-13T13:25:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_2 128M 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:25:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:25:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:25:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2024-04-13T13:25:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:25:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-04-13T13:25:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:25:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-04-13T13:25:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:25:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_2': 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 128a6d8fd1ef 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.128a6d8fd1ef 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:22 2024 2024-04-13T13:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:22 2024 2024-04-13T13:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:22 2024 2024-04-13T13:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 45670160-3031-494c-bc36-3e5d858b75c1 2024-04-13T13:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:25:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:25:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-13T13:25:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:25:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:25:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=7 2024-04-13T13:25:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:25:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:25:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 41198592 bytes 2024-04-13T13:25:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 110326784 bytes 2024-04-13T13:25:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 124001280 bytes 2024-04-13T13:25:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 124608512 bytes 2024-04-13T13:25:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 147020800 bytes 2024-04-13T13:25:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 206612480 bytes 2024-04-13T13:25:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 281954304 bytes 2024-04-13T13:25:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 313763840 bytes 2024-04-13T13:25:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 342859776 bytes 2024-04-13T13:25:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 370627584 bytes 2024-04-13T13:25:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 376720384 bytes 2024-04-13T13:25:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 404279296 bytes 2024-04-13T13:25:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 450867200 bytes 2024-04-13T13:25:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 480609280 bytes 2024-04-13T13:25:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 499174400 bytes 2024-04-13T13:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 499775488 bytes 2024-04-13T13:25:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 553542656 bytes 2024-04-13T13:25:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 563308544 bytes 2024-04-13T13:25:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 605209600 bytes 2024-04-13T13:25:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 649279488 bytes 2024-04-13T13:25:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 663496704 bytes 2024-04-13T13:25:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 675281920 bytes 2024-04-13T13:25:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 748855296 bytes 2024-04-13T13:25:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 749494272 bytes 2024-04-13T13:25:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 799924224 bytes 2024-04-13T13:25:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 889164800 bytes 2024-04-13T13:25:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 896002048 bytes 2024-04-13T13:25:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 914040832 bytes 2024-04-13T13:25:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 929105920 bytes 2024-04-13T13:25:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1045301248 bytes 2024-04-13T13:25:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1066682368 bytes 2024-04-13T13:25:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1095483392 bytes 2024-04-13T13:25:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1163572224 bytes 2024-04-13T13:25:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1222719488 bytes 2024-04-13T13:25:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1275059200 bytes 2024-04-13T13:25:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1340275712 bytes 2024-04-13T13:25:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1396562944 bytes 2024-04-13T13:25:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1428179968 bytes 2024-04-13T13:25:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1439917056 bytes 2024-04-13T13:25:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1481372672 bytes 2024-04-13T13:25:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1543059456 bytes 2024-04-13T13:25:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1577674752 bytes 2024-04-13T13:25:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1615975424 bytes 2024-04-13T13:25:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1685165056 bytes 2024-04-13T13:25:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1724165120 bytes 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_3 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_3 128M 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:25:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:25:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:25:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:25:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2024-04-13T13:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-04-13T13:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-04-13T13:25:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:25:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_3': 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1292d408538f 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1292d408538f 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:25:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:25:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:30 2024 2024-04-13T13:25:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:30 2024 2024-04-13T13:25:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:30 2024 2024-04-13T13:25:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:25:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:25:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 173021c1-45fc-4ddf-ae55-e453164e52a8 2024-04-13T13:25:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:25:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:25:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-13T13:25:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:25:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:25:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=14 2024-04-13T13:25:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:25:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:25:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 67486720 bytes 2024-04-13T13:25:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 127910912 bytes 2024-04-13T13:25:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 139169792 bytes 2024-04-13T13:25:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 197373952 bytes 2024-04-13T13:25:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 252759040 bytes 2024-04-13T13:25:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 300395520 bytes 2024-04-13T13:25:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 320295936 bytes 2024-04-13T13:25:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 323125248 bytes 2024-04-13T13:25:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 398273536 bytes 2024-04-13T13:25:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 466843648 bytes 2024-04-13T13:25:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 530775040 bytes 2024-04-13T13:25:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 548636672 bytes 2024-04-13T13:25:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 579170304 bytes 2024-04-13T13:25:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 653619200 bytes 2024-04-13T13:25:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 666703872 bytes 2024-04-13T13:25:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 754270208 bytes 2024-04-13T13:25:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 807296000 bytes 2024-04-13T13:25:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 831660032 bytes 2024-04-13T13:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 875082752 bytes 2024-04-13T13:25:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 922260480 bytes 2024-04-13T13:25:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 922880000 bytes 2024-04-13T13:25:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 937541632 bytes 2024-04-13T13:25:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 957320192 bytes 2024-04-13T13:25:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 957924352 bytes 2024-04-13T13:25:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1043957760 bytes 2024-04-13T13:25:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1097859072 bytes 2024-04-13T13:25:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1117441024 bytes 2024-04-13T13:25:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1160384512 bytes 2024-04-13T13:25:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1173777408 bytes 2024-04-13T13:25:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1260636160 bytes 2024-04-13T13:25:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1286609920 bytes 2024-04-13T13:25:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1376189440 bytes 2024-04-13T13:25:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1392739328 bytes 2024-04-13T13:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1481024512 bytes 2024-04-13T13:25:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1536556032 bytes 2024-04-13T13:25:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1609360384 bytes 2024-04-13T13:25:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1615190016 bytes 2024-04-13T13:25:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1628744704 bytes 2024-04-13T13:25:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1662781440 bytes 2024-04-13T13:25:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1699188736 bytes 2024-04-13T13:25:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1763076096 bytes 2024-04-13T13:25:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1822758912 bytes 2024-04-13T13:25:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1841804288 bytes 2024-04-13T13:25:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1875173376 bytes 2024-04-13T13:25:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1923258368 bytes 2024-04-13T13:25:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1946981376 bytes 2024-04-13T13:25:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1975745536 bytes 2024-04-13T13:25:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2000582656 bytes 2024-04-13T13:25:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2055702528 bytes 2024-04-13T13:25:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2081541120 bytes 2024-04-13T13:25:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2200867840 bytes 2024-04-13T13:25:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2286745600 bytes 2024-04-13T13:25:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2311749632 bytes 2024-04-13T13:25:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2369312768 bytes 2024-04-13T13:25:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2402818048 bytes 2024-04-13T13:25:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2493926400 bytes 2024-04-13T13:25:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2526914560 bytes 2024-04-13T13:25:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2577106944 bytes 2024-04-13T13:25:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2645642240 bytes 2024-04-13T13:25:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2666360832 bytes 2024-04-13T13:25:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2671056896 bytes 2024-04-13T13:25:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2710983680 bytes 2024-04-13T13:25:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2768807936 bytes 2024-04-13T13:25:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2811176960 bytes 2024-04-13T13:25:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2870498304 bytes 2024-04-13T13:25:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2938452992 bytes 2024-04-13T13:25:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2942575616 bytes 2024-04-13T13:25:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2960020480 bytes 2024-04-13T13:25:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2999613440 bytes 2024-04-13T13:25:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 3039781888 bytes 2024-04-13T13:25:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 3047086080 bytes 2024-04-13T13:25:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 3077421056 bytes 2024-04-13T13:25:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 3113055232 bytes 2024-04-13T13:25:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 3116413952 bytes 2024-04-13T13:25:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 3149664256 bytes 2024-04-13T13:25:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3152539648 bytes 2024-04-13T13:25:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 3178476544 bytes 2024-04-13T13:25:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7800 IOs, 3217414144 bytes 2024-04-13T13:25:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7900 IOs, 3278502912 bytes 2024-04-13T13:25:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8000 IOs, 3328721920 bytes 2024-04-13T13:25:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8100 IOs, 3406967808 bytes 2024-04-13T13:25:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8200 IOs, 3463949312 bytes 2024-04-13T13:25:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8300 IOs, 3522870272 bytes 2024-04-13T13:25:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8400 IOs, 3551394816 bytes 2024-04-13T13:25:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8500 IOs, 3649915904 bytes 2024-04-13T13:25:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8600 IOs, 3720053760 bytes 2024-04-13T13:25:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_4 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_4 128M 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2024-04-13T13:25:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:25:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-04-13T13:25:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:25:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-04-13T13:25:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:25:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_4': 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 129aa199ecc4 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.129aa199ecc4 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:25:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:44 2024 2024-04-13T13:25:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:44 2024 2024-04-13T13:25:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:44 2024 2024-04-13T13:25:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:25:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:25:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8d69fb4b-4707-45d2-bd35-f5dc2bafa56b 2024-04-13T13:25:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:25:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:25:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-13T13:25:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:25:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:25:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=10 2024-04-13T13:25:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:25:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:25:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 34523136 bytes 2024-04-13T13:25:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 142407680 bytes 2024-04-13T13:25:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 155833344 bytes 2024-04-13T13:25:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 269167616 bytes 2024-04-13T13:25:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 323091456 bytes 2024-04-13T13:25:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 367560704 bytes 2024-04-13T13:25:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 392819712 bytes 2024-04-13T13:25:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 442184704 bytes 2024-04-13T13:25:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 480624640 bytes 2024-04-13T13:25:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 522240000 bytes 2024-04-13T13:25:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 525260800 bytes 2024-04-13T13:25:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 550743040 bytes 2024-04-13T13:25:47.839 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:25:47.844+0000 7f59186a3640 -1 reset not still connected to 0x55ea1741f380 2024-04-13T13:25:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 610952192 bytes 2024-04-13T13:25:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 667973632 bytes 2024-04-13T13:25:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 740243456 bytes 2024-04-13T13:25:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 760699904 bytes 2024-04-13T13:25:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 810506240 bytes 2024-04-13T13:25:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 828705792 bytes 2024-04-13T13:25:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 867292160 bytes 2024-04-13T13:25:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 911864832 bytes 2024-04-13T13:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 976431104 bytes 2024-04-13T13:25:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1020025856 bytes 2024-04-13T13:25:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1053400064 bytes 2024-04-13T13:25:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1057647616 bytes 2024-04-13T13:25:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1077603328 bytes 2024-04-13T13:25:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1182276608 bytes 2024-04-13T13:25:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1246435328 bytes 2024-04-13T13:25:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1262406656 bytes 2024-04-13T13:25:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1289446400 bytes 2024-04-13T13:25:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1355162624 bytes 2024-04-13T13:25:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1376934912 bytes 2024-04-13T13:25:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1397084160 bytes 2024-04-13T13:25:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1407548416 bytes 2024-04-13T13:25:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1441172480 bytes 2024-04-13T13:25:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1452030976 bytes 2024-04-13T13:25:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1456645120 bytes 2024-04-13T13:25:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1508152320 bytes 2024-04-13T13:25:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1526940672 bytes 2024-04-13T13:25:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1567774720 bytes 2024-04-13T13:25:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1621210112 bytes 2024-04-13T13:25:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1662286848 bytes 2024-04-13T13:25:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1762244608 bytes 2024-04-13T13:25:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1800693760 bytes 2024-04-13T13:25:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1843818496 bytes 2024-04-13T13:25:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1858626560 bytes 2024-04-13T13:25:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1886347264 bytes 2024-04-13T13:25:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1941549056 bytes 2024-04-13T13:25:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1951284224 bytes 2024-04-13T13:25:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1973584896 bytes 2024-04-13T13:25:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2034025472 bytes 2024-04-13T13:25:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2068995072 bytes 2024-04-13T13:25:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2128962560 bytes 2024-04-13T13:25:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2152498176 bytes 2024-04-13T13:25:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2203800576 bytes 2024-04-13T13:25:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2243384320 bytes 2024-04-13T13:25:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2266369024 bytes 2024-04-13T13:25:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2284680192 bytes 2024-04-13T13:25:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2297501696 bytes 2024-04-13T13:25:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2339051520 bytes 2024-04-13T13:25:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2403186688 bytes 2024-04-13T13:25:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2459623424 bytes 2024-04-13T13:25:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2485671936 bytes 2024-04-13T13:25:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2517367808 bytes 2024-04-13T13:25:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:25:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_5 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_5 128M 2024-04-13T13:25:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:25:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:25:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:25:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:25:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:25:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2024-04-13T13:25:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:25:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-04-13T13:25:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:25:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:25:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-04-13T13:25:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:25:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_5': 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12a2e7bcb974 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12a2e7bcb974 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:55 2024 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:55 2024 2024-04-13T13:25:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:55 2024 2024-04-13T13:25:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:25:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:25:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: b72ab703-5a35-4d41-90bc-db2c081ebc57 2024-04-13T13:25:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:25:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:25:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-13T13:25:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:25:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:25:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:25:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:25:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=14 2024-04-13T13:25:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:25:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:25:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 33542144 bytes 2024-04-13T13:25:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 60331008 bytes 2024-04-13T13:25:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 82919424 bytes 2024-04-13T13:25:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 115548160 bytes 2024-04-13T13:25:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 141337600 bytes 2024-04-13T13:25:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 184040448 bytes 2024-04-13T13:25:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 242465792 bytes 2024-04-13T13:25:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 296496128 bytes 2024-04-13T13:25:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 344519680 bytes 2024-04-13T13:25:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 399144960 bytes 2024-04-13T13:25:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 444345344 bytes 2024-04-13T13:25:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 460821504 bytes 2024-04-13T13:25:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 500042752 bytes 2024-04-13T13:25:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 560922624 bytes 2024-04-13T13:25:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 614063104 bytes 2024-04-13T13:25:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 638698496 bytes 2024-04-13T13:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 709268480 bytes 2024-04-13T13:25:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 714187776 bytes 2024-04-13T13:25:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 748080128 bytes 2024-04-13T13:25:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 757666816 bytes 2024-04-13T13:25:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 822383616 bytes 2024-04-13T13:26:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 834902016 bytes 2024-04-13T13:26:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 862745600 bytes 2024-04-13T13:26:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 864635904 bytes 2024-04-13T13:26:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 889764864 bytes 2024-04-13T13:26:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 920961024 bytes 2024-04-13T13:26:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1005090816 bytes 2024-04-13T13:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1028691968 bytes 2024-04-13T13:26:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1080980480 bytes 2024-04-13T13:26:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1180027904 bytes 2024-04-13T13:26:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1183717376 bytes 2024-04-13T13:26:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1272730624 bytes 2024-04-13T13:26:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1303593984 bytes 2024-04-13T13:26:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1390359552 bytes 2024-04-13T13:26:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1435919360 bytes 2024-04-13T13:26:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1515812864 bytes 2024-04-13T13:26:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1542872064 bytes 2024-04-13T13:26:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1641407488 bytes 2024-04-13T13:26:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1657758720 bytes 2024-04-13T13:26:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1681816576 bytes 2024-04-13T13:26:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1742572544 bytes 2024-04-13T13:26:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1769341952 bytes 2024-04-13T13:26:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1800365056 bytes 2024-04-13T13:26:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1872614400 bytes 2024-04-13T13:26:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1880360960 bytes 2024-04-13T13:26:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1895470080 bytes 2024-04-13T13:26:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1936095232 bytes 2024-04-13T13:26:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1958029312 bytes 2024-04-13T13:26:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1983443968 bytes 2024-04-13T13:26:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2041676800 bytes 2024-04-13T13:26:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2145812480 bytes 2024-04-13T13:26:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2192772096 bytes 2024-04-13T13:26:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2216469504 bytes 2024-04-13T13:26:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2263172096 bytes 2024-04-13T13:26:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2340940800 bytes 2024-04-13T13:26:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2359152640 bytes 2024-04-13T13:26:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2380912640 bytes 2024-04-13T13:26:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2397285376 bytes 2024-04-13T13:26:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2398200832 bytes 2024-04-13T13:26:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2417816576 bytes 2024-04-13T13:26:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2455625728 bytes 2024-04-13T13:26:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2492515328 bytes 2024-04-13T13:26:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2535128064 bytes 2024-04-13T13:26:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2554295296 bytes 2024-04-13T13:26:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2606615552 bytes 2024-04-13T13:26:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2630630400 bytes 2024-04-13T13:26:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2647657472 bytes 2024-04-13T13:26:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2672306176 bytes 2024-04-13T13:26:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2701569024 bytes 2024-04-13T13:26:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2772507648 bytes 2024-04-13T13:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2804255744 bytes 2024-04-13T13:26:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2856228864 bytes 2024-04-13T13:26:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2860807168 bytes 2024-04-13T13:26:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 2861417472 bytes 2024-04-13T13:26:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 2961499136 bytes 2024-04-13T13:26:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3037043712 bytes 2024-04-13T13:26:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 3071294464 bytes 2024-04-13T13:26:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7800 IOs, 3105350656 bytes 2024-04-13T13:26:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7900 IOs, 3141473280 bytes 2024-04-13T13:26:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8000 IOs, 3165498368 bytes 2024-04-13T13:26:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8100 IOs, 3229847552 bytes 2024-04-13T13:26:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8200 IOs, 3244449792 bytes 2024-04-13T13:26:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8300 IOs, 3255135232 bytes 2024-04-13T13:26:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8400 IOs, 3272824832 bytes 2024-04-13T13:26:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8500 IOs, 3312385024 bytes 2024-04-13T13:26:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8600 IOs, 3355340800 bytes 2024-04-13T13:26:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8700 IOs, 3373129728 bytes 2024-04-13T13:26:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8800 IOs, 3417600000 bytes 2024-04-13T13:26:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8900 IOs, 3440553984 bytes 2024-04-13T13:26:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9000 IOs, 3441173504 bytes 2024-04-13T13:26:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:26:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:26:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:26:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:26:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:26:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_6 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_6 128M 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:26:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:26:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:26:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:26:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2024-04-13T13:26:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:26:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-04-13T13:26:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:26:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-04-13T13:26:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:26:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_6': 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12aa9fc971e0 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12aa9fc971e0 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:26:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:26:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:26:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:10 2024 2024-04-13T13:26:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:10 2024 2024-04-13T13:26:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:10 2024 2024-04-13T13:26:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:26:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:26:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1466461b-b5d7-4bb3-9928-7a3a2fd5b22f 2024-04-13T13:26:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:26:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:26:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-13T13:26:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:26:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:26:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=7 2024-04-13T13:26:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:26:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:26:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 32909312 bytes 2024-04-13T13:26:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 74182656 bytes 2024-04-13T13:26:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 127634432 bytes 2024-04-13T13:26:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 161032192 bytes 2024-04-13T13:26:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 216915968 bytes 2024-04-13T13:26:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 259690496 bytes 2024-04-13T13:26:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 275347456 bytes 2024-04-13T13:26:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 313817088 bytes 2024-04-13T13:26:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 383197184 bytes 2024-04-13T13:26:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 405151744 bytes 2024-04-13T13:26:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 425350144 bytes 2024-04-13T13:26:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 509947904 bytes 2024-04-13T13:26:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 588217344 bytes 2024-04-13T13:26:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 643422208 bytes 2024-04-13T13:26:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 661175296 bytes 2024-04-13T13:26:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 694268928 bytes 2024-04-13T13:26:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 760139776 bytes 2024-04-13T13:26:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 815584256 bytes 2024-04-13T13:26:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 857218048 bytes 2024-04-13T13:26:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 950832128 bytes 2024-04-13T13:26:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 957661184 bytes 2024-04-13T13:26:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 998484992 bytes 2024-04-13T13:26:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1042484224 bytes 2024-04-13T13:26:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1056317440 bytes 2024-04-13T13:26:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1103101952 bytes 2024-04-13T13:26:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1135808512 bytes 2024-04-13T13:26:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1194616832 bytes 2024-04-13T13:26:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1197786112 bytes 2024-04-13T13:26:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1199832064 bytes 2024-04-13T13:26:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1262054400 bytes 2024-04-13T13:26:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1315518464 bytes 2024-04-13T13:26:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1342145536 bytes 2024-04-13T13:26:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1368975360 bytes 2024-04-13T13:26:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1372895232 bytes 2024-04-13T13:26:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1389369344 bytes 2024-04-13T13:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1427973120 bytes 2024-04-13T13:26:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1482425344 bytes 2024-04-13T13:26:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1524555776 bytes 2024-04-13T13:26:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1582576640 bytes 2024-04-13T13:26:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1603475456 bytes 2024-04-13T13:26:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1631905792 bytes 2024-04-13T13:26:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1669420032 bytes 2024-04-13T13:26:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1695891456 bytes 2024-04-13T13:26:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_7 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:26:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:26:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_7 128M 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2024-04-13T13:26:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:26:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-04-13T13:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-04-13T13:26:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:26:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-04-13T13:26:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_7': 2024-04-13T13:26:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:26:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:26:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:26:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b4660b3fc7 2024-04-13T13:26:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b4660b3fc7 2024-04-13T13:26:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:26:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:26:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:26:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:26:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:18 2024 2024-04-13T13:26:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:18 2024 2024-04-13T13:26:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:18 2024 2024-04-13T13:26:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:26:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:26:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 5e948815-4bc4-4300-914c-1855c84e9f33 2024-04-13T13:26:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:26:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:26:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-13T13:26:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:26:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:26:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:26:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:26:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:26:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 42568704 bytes 2024-04-13T13:26:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 115107840 bytes 2024-04-13T13:26:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 140324864 bytes 2024-04-13T13:26:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 161436672 bytes 2024-04-13T13:26:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 201730048 bytes 2024-04-13T13:26:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 240398336 bytes 2024-04-13T13:26:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 289020928 bytes 2024-04-13T13:26:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 311017472 bytes 2024-04-13T13:26:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 327485440 bytes 2024-04-13T13:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 343255040 bytes 2024-04-13T13:26:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 360285184 bytes 2024-04-13T13:26:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 362749952 bytes 2024-04-13T13:26:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 432168960 bytes 2024-04-13T13:26:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 508977152 bytes 2024-04-13T13:26:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 556700672 bytes 2024-04-13T13:26:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 589849600 bytes 2024-04-13T13:26:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 624028672 bytes 2024-04-13T13:26:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 657857536 bytes 2024-04-13T13:26:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 744534016 bytes 2024-04-13T13:26:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 797159424 bytes 2024-04-13T13:26:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 871601152 bytes 2024-04-13T13:26:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 872217600 bytes 2024-04-13T13:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 920822784 bytes 2024-04-13T13:26:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 961883136 bytes 2024-04-13T13:26:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 973536256 bytes 2024-04-13T13:26:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1034524672 bytes 2024-04-13T13:26:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1100679168 bytes 2024-04-13T13:26:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1153661952 bytes 2024-04-13T13:26:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1228961792 bytes 2024-04-13T13:26:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1258665984 bytes 2024-04-13T13:26:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1278230528 bytes 2024-04-13T13:26:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1287805952 bytes 2024-04-13T13:26:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1341193216 bytes 2024-04-13T13:26:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1362365440 bytes 2024-04-13T13:26:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1409453056 bytes 2024-04-13T13:26:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1466241024 bytes 2024-04-13T13:26:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1512992768 bytes 2024-04-13T13:26:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1519009792 bytes 2024-04-13T13:26:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1580486656 bytes 2024-04-13T13:26:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1599984640 bytes 2024-04-13T13:26:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1636076544 bytes 2024-04-13T13:26:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1684669440 bytes 2024-04-13T13:26:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1748769792 bytes 2024-04-13T13:26:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1751149568 bytes 2024-04-13T13:26:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1787458560 bytes 2024-04-13T13:26:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1822715904 bytes 2024-04-13T13:26:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1872970752 bytes 2024-04-13T13:26:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1887950848 bytes 2024-04-13T13:26:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1916635136 bytes 2024-04-13T13:26:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1917254656 bytes 2024-04-13T13:26:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1954280448 bytes 2024-04-13T13:26:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2013770752 bytes 2024-04-13T13:26:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2035951616 bytes 2024-04-13T13:26:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2115866624 bytes 2024-04-13T13:26:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2142153728 bytes 2024-04-13T13:26:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2201112576 bytes 2024-04-13T13:26:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2237685760 bytes 2024-04-13T13:26:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2238286848 bytes 2024-04-13T13:26:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2257592320 bytes 2024-04-13T13:26:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2292042752 bytes 2024-04-13T13:26:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2304326656 bytes 2024-04-13T13:26:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2332253184 bytes 2024-04-13T13:26:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2404825088 bytes 2024-04-13T13:26:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2433543168 bytes 2024-04-13T13:26:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2536443904 bytes 2024-04-13T13:26:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2638949376 bytes 2024-04-13T13:26:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2676847616 bytes 2024-04-13T13:26:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2708592640 bytes 2024-04-13T13:26:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2710071296 bytes 2024-04-13T13:26:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2711028736 bytes 2024-04-13T13:26:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2739518464 bytes 2024-04-13T13:26:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2754190336 bytes 2024-04-13T13:26:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2755649536 bytes 2024-04-13T13:26:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 2830120960 bytes 2024-04-13T13:26:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 2901071872 bytes 2024-04-13T13:26:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 2904833024 bytes 2024-04-13T13:26:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 2990529536 bytes 2024-04-13T13:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_8 2024-04-13T13:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-04-13T13:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_8 128M 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:26:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:26:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:26:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:26:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2024-04-13T13:26:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:26:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-04-13T13:26:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:26:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-04-13T13:26:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:26:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-04-13T13:26:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_8': 2024-04-13T13:26:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12beb019044c 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12beb019044c 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:31 2024 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:31 2024 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:31 2024 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:26:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:26:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 970b3a1e-6448-44f1-be2c-e3b2bc2b08af 2024-04-13T13:26:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-13T13:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:26:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=6 2024-04-13T13:26:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:26:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:26:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 44923904 bytes 2024-04-13T13:26:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 112561152 bytes 2024-04-13T13:26:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 138793984 bytes 2024-04-13T13:26:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 157251584 bytes 2024-04-13T13:26:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 157848576 bytes 2024-04-13T13:26:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 210772992 bytes 2024-04-13T13:26:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 271801344 bytes 2024-04-13T13:26:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 301416448 bytes 2024-04-13T13:26:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 348717056 bytes 2024-04-13T13:26:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 443808768 bytes 2024-04-13T13:26:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 586516480 bytes 2024-04-13T13:26:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 649199616 bytes 2024-04-13T13:26:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 683842560 bytes 2024-04-13T13:26:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 732150784 bytes 2024-04-13T13:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 826829824 bytes 2024-04-13T13:26:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 827419648 bytes 2024-04-13T13:26:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 886305792 bytes 2024-04-13T13:26:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 920172544 bytes 2024-04-13T13:26:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 920790016 bytes 2024-04-13T13:26:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 942708736 bytes 2024-04-13T13:26:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 986586112 bytes 2024-04-13T13:26:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1019787264 bytes 2024-04-13T13:26:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1036990464 bytes 2024-04-13T13:26:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1053185024 bytes 2024-04-13T13:26:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1112114176 bytes 2024-04-13T13:26:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1149192192 bytes 2024-04-13T13:26:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1171781632 bytes 2024-04-13T13:26:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1184173056 bytes 2024-04-13T13:26:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1224219648 bytes 2024-04-13T13:26:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1230256128 bytes 2024-04-13T13:26:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1283554304 bytes 2024-04-13T13:26:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1284182016 bytes 2024-04-13T13:26:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1312287744 bytes 2024-04-13T13:26:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1365698560 bytes 2024-04-13T13:26:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1368117248 bytes 2024-04-13T13:26:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1408285696 bytes 2024-04-13T13:26:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1476412416 bytes 2024-04-13T13:26:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:26:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:26:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:26:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:26:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_9 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_9 128M 2024-04-13T13:26:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2024-04-13T13:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-04-13T13:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:26:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-04-13T13:26:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:26:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_9': 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c7b0401a0f 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c7b0401a0f 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:39 2024 2024-04-13T13:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:39 2024 2024-04-13T13:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:39 2024 2024-04-13T13:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 9b0f1c92-41a8-4559-8a23-3818c789553a 2024-04-13T13:26:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:26:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:26:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-13T13:26:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:26:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:26:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=13 2024-04-13T13:26:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:26:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:26:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 1612800 bytes 2024-04-13T13:26:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 40365056 bytes 2024-04-13T13:26:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 64875520 bytes 2024-04-13T13:26:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 106390528 bytes 2024-04-13T13:26:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 112321536 bytes 2024-04-13T13:26:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 130818048 bytes 2024-04-13T13:26:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 196137984 bytes 2024-04-13T13:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 237544448 bytes 2024-04-13T13:26:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 289310720 bytes 2024-04-13T13:26:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 326822912 bytes 2024-04-13T13:26:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 327453696 bytes 2024-04-13T13:26:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 418982912 bytes 2024-04-13T13:26:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 450238464 bytes 2024-04-13T13:26:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 524270592 bytes 2024-04-13T13:26:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 527193088 bytes 2024-04-13T13:26:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 551610368 bytes 2024-04-13T13:26:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 581267456 bytes 2024-04-13T13:26:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 631320576 bytes 2024-04-13T13:26:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 669748224 bytes 2024-04-13T13:26:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 713289728 bytes 2024-04-13T13:26:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 741831680 bytes 2024-04-13T13:26:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 772525056 bytes 2024-04-13T13:26:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 815932416 bytes 2024-04-13T13:26:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 836756480 bytes 2024-04-13T13:26:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 837367808 bytes 2024-04-13T13:26:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 854832128 bytes 2024-04-13T13:26:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 868877312 bytes 2024-04-13T13:26:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 892177408 bytes 2024-04-13T13:26:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 911784960 bytes 2024-04-13T13:26:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 997813248 bytes 2024-04-13T13:26:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1027722240 bytes 2024-04-13T13:26:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1063588864 bytes 2024-04-13T13:26:44.813 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7861627808939836) > weight (0.25) 2024-04-13T13:26:44.813 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5922719095798746) > weight (0.25) 2024-04-13T13:26:44.813 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4980325837641375) > weight (0.25) 2024-04-13T13:26:44.814 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6699660312077983) > weight (0.25) 2024-04-13T13:26:44.814 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-04-13T13:26:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1091211264 bytes 2024-04-13T13:26:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1117820928 bytes 2024-04-13T13:26:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1156678656 bytes 2024-04-13T13:26:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1196518400 bytes 2024-04-13T13:26:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1230525440 bytes 2024-04-13T13:26:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1365066752 bytes 2024-04-13T13:26:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1383215104 bytes 2024-04-13T13:26:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1393104896 bytes 2024-04-13T13:26:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1451979776 bytes 2024-04-13T13:26:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1481110528 bytes 2024-04-13T13:26:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1516323840 bytes 2024-04-13T13:26:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1540721664 bytes 2024-04-13T13:26:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1546726400 bytes 2024-04-13T13:26:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1573738496 bytes 2024-04-13T13:26:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1645205504 bytes 2024-04-13T13:26:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1684899840 bytes 2024-04-13T13:26:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1762416640 bytes 2024-04-13T13:26:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1809029120 bytes 2024-04-13T13:26:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1916643328 bytes 2024-04-13T13:26:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1963751424 bytes 2024-04-13T13:26:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 1995116544 bytes 2024-04-13T13:26:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2011373568 bytes 2024-04-13T13:26:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2048080896 bytes 2024-04-13T13:26:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2116176896 bytes 2024-04-13T13:26:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2191982592 bytes 2024-04-13T13:26:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2253294592 bytes 2024-04-13T13:26:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2288306176 bytes 2024-04-13T13:26:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2294768640 bytes 2024-04-13T13:26:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2320724992 bytes 2024-04-13T13:26:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2445834240 bytes 2024-04-13T13:26:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2514369536 bytes 2024-04-13T13:26:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2522795008 bytes 2024-04-13T13:26:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2555273216 bytes 2024-04-13T13:26:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2583289856 bytes 2024-04-13T13:26:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2588151808 bytes 2024-04-13T13:26:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2613306368 bytes 2024-04-13T13:26:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2667150336 bytes 2024-04-13T13:26:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2667759616 bytes 2024-04-13T13:26:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2749491200 bytes 2024-04-13T13:26:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2767538176 bytes 2024-04-13T13:26:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2775390208 bytes 2024-04-13T13:26:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 2864072704 bytes 2024-04-13T13:26:51.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 2893936640 bytes 2024-04-13T13:26:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 2930539520 bytes 2024-04-13T13:26:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 3001220096 bytes 2024-04-13T13:26:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7800 IOs, 3016207360 bytes 2024-04-13T13:26:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7900 IOs, 3064203264 bytes 2024-04-13T13:26:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8000 IOs, 3092348928 bytes 2024-04-13T13:26:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8100 IOs, 3139290112 bytes 2024-04-13T13:26:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8200 IOs, 3139908608 bytes 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_10 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_10 128M 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:26:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:26:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:26:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:26:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:26:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2024-04-13T13:26:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:26:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-04-13T13:26:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:26:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:26:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-04-13T13:26:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:26:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-04-13T13:26:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_10': 2024-04-13T13:26:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:26:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:26:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:26:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12d12a92a84a 2024-04-13T13:26:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12d12a92a84a 2024-04-13T13:26:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:53 2024 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:53 2024 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:53 2024 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 7cb77a9f-0f4a-4ba9-bc4b-ae0678719819 2024-04-13T13:26:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:26:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:26:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-13T13:26:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:26:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:26:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:26:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:26:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=7 2024-04-13T13:26:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:26:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:26:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 34719744 bytes 2024-04-13T13:26:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 60947456 bytes 2024-04-13T13:26:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 89359360 bytes 2024-04-13T13:26:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 106723328 bytes 2024-04-13T13:26:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 144377856 bytes 2024-04-13T13:26:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 189160448 bytes 2024-04-13T13:26:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 201141248 bytes 2024-04-13T13:26:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 228951040 bytes 2024-04-13T13:26:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 240657408 bytes 2024-04-13T13:26:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 346654720 bytes 2024-04-13T13:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 412021760 bytes 2024-04-13T13:26:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 471619584 bytes 2024-04-13T13:26:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 479529984 bytes 2024-04-13T13:26:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 513213440 bytes 2024-04-13T13:26:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 513838080 bytes 2024-04-13T13:26:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 598552576 bytes 2024-04-13T13:26:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 637138944 bytes 2024-04-13T13:26:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 716291072 bytes 2024-04-13T13:26:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 772118528 bytes 2024-04-13T13:26:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 784953344 bytes 2024-04-13T13:26:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 804794368 bytes 2024-04-13T13:26:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 810571776 bytes 2024-04-13T13:26:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 842158080 bytes 2024-04-13T13:26:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 945482752 bytes 2024-04-13T13:26:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 946076672 bytes 2024-04-13T13:26:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1001134080 bytes 2024-04-13T13:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1030666240 bytes 2024-04-13T13:26:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1058363392 bytes 2024-04-13T13:26:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1083405312 bytes 2024-04-13T13:26:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1199367168 bytes 2024-04-13T13:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1230912512 bytes 2024-04-13T13:26:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1249166336 bytes 2024-04-13T13:26:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1371658240 bytes 2024-04-13T13:26:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1426324480 bytes 2024-04-13T13:26:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1453075456 bytes 2024-04-13T13:26:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1491635200 bytes 2024-04-13T13:27:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1532822528 bytes 2024-04-13T13:27:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1638810624 bytes 2024-04-13T13:27:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1681634304 bytes 2024-04-13T13:27:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1698501632 bytes 2024-04-13T13:27:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1766810624 bytes 2024-04-13T13:27:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1804760064 bytes 2024-04-13T13:27:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:27:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:27:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:27:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:27:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:27:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_11 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_11 128M 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2024-04-13T13:27:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:27:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-04-13T13:27:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:27:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-04-13T13:27:01.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8486030770472859) > weight (0.25) 2024-04-13T13:27:01.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5959043009018001) > weight (0.25) 2024-04-13T13:27:01.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4886509562047249) > weight (0.25) 2024-04-13T13:27:01.711 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9268905665060897) > weight (0.25) 2024-04-13T13:27:01.711 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-04-13T13:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:27:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-04-13T13:27:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_11': 2024-04-13T13:27:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12da83cb974a 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12da83cb974a 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:01 2024 2024-04-13T13:27:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:01 2024 2024-04-13T13:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:01 2024 2024-04-13T13:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 59ed4044-a6c7-40ba-a8ff-0113c71c89da 2024-04-13T13:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_11 2024-04-13T13:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=6 2024-04-13T13:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:27:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 37419008 bytes 2024-04-13T13:27:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 91123712 bytes 2024-04-13T13:27:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 128114688 bytes 2024-04-13T13:27:02.694 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:27:02.699+0000 7fd27b621640 -1 reset not still connected to 0x55cbcaf149c0 2024-04-13T13:27:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 128844800 bytes 2024-04-13T13:27:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 209656832 bytes 2024-04-13T13:27:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 226250752 bytes 2024-04-13T13:27:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 226860032 bytes 2024-04-13T13:27:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 285245440 bytes 2024-04-13T13:27:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 340392960 bytes 2024-04-13T13:27:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 382431232 bytes 2024-04-13T13:27:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 384327680 bytes 2024-04-13T13:27:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 434076672 bytes 2024-04-13T13:27:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 478901248 bytes 2024-04-13T13:27:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 501286912 bytes 2024-04-13T13:27:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 545774592 bytes 2024-04-13T13:27:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 613649408 bytes 2024-04-13T13:27:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 654280704 bytes 2024-04-13T13:27:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 728978432 bytes 2024-04-13T13:27:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 771818496 bytes 2024-04-13T13:27:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 822168576 bytes 2024-04-13T13:27:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 847493120 bytes 2024-04-13T13:27:05.576 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:27:05.581+0000 7fd27b621640 -1 reset not still connected to 0x55cbcaf149c0 2024-04-13T13:27:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 868207616 bytes 2024-04-13T13:27:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 884146176 bytes 2024-04-13T13:27:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 989634560 bytes 2024-04-13T13:27:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1010323456 bytes 2024-04-13T13:27:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1071352832 bytes 2024-04-13T13:27:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1123257344 bytes 2024-04-13T13:27:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1145997312 bytes 2024-04-13T13:27:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1198728192 bytes 2024-04-13T13:27:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1213147136 bytes 2024-04-13T13:27:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1226190848 bytes 2024-04-13T13:27:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1258757120 bytes 2024-04-13T13:27:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1287078912 bytes 2024-04-13T13:27:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1298997248 bytes 2024-04-13T13:27:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1343267840 bytes 2024-04-13T13:27:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1389934592 bytes 2024-04-13T13:27:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1448389632 bytes 2024-04-13T13:27:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:27:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_12 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_12 128M 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2024-04-13T13:27:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:27:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-04-13T13:27:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:27:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-04-13T13:27:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:27:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-04-13T13:27:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_12': 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12e2fd14ccc0 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12e2fd14ccc0 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:08 2024 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:08 2024 2024-04-13T13:27:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:08 2024 2024-04-13T13:27:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:27:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:27:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 5b84301a-f350-4908-84da-ddb3ac371259 2024-04-13T13:27:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:27:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:27:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_12 2024-04-13T13:27:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:27:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:27:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=7 2024-04-13T13:27:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:27:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:27:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 89940992 bytes 2024-04-13T13:27:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 191523840 bytes 2024-04-13T13:27:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 202811392 bytes 2024-04-13T13:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 220849152 bytes 2024-04-13T13:27:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 239657984 bytes 2024-04-13T13:27:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 255345664 bytes 2024-04-13T13:27:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 315141120 bytes 2024-04-13T13:27:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 349648896 bytes 2024-04-13T13:27:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 376799232 bytes 2024-04-13T13:27:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 406069248 bytes 2024-04-13T13:27:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 470723584 bytes 2024-04-13T13:27:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 482151424 bytes 2024-04-13T13:27:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 549309440 bytes 2024-04-13T13:27:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 587813888 bytes 2024-04-13T13:27:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 641697792 bytes 2024-04-13T13:27:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 754924544 bytes 2024-04-13T13:27:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 755566592 bytes 2024-04-13T13:27:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 830548992 bytes 2024-04-13T13:27:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 879642624 bytes 2024-04-13T13:27:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 949587968 bytes 2024-04-13T13:27:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 953200640 bytes 2024-04-13T13:27:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1016947712 bytes 2024-04-13T13:27:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1049494528 bytes 2024-04-13T13:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1076495360 bytes 2024-04-13T13:27:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1094257664 bytes 2024-04-13T13:27:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1128608768 bytes 2024-04-13T13:27:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1161546752 bytes 2024-04-13T13:27:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1165465600 bytes 2024-04-13T13:27:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1204477952 bytes 2024-04-13T13:27:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1253550080 bytes 2024-04-13T13:27:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1328874496 bytes 2024-04-13T13:27:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1424280576 bytes 2024-04-13T13:27:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1480055808 bytes 2024-04-13T13:27:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1555143680 bytes 2024-04-13T13:27:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1694998528 bytes 2024-04-13T13:27:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1715213312 bytes 2024-04-13T13:27:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1715832832 bytes 2024-04-13T13:27:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1837350912 bytes 2024-04-13T13:27:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1863891968 bytes 2024-04-13T13:27:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:27:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:27:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_13 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_13 128M 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2024-04-13T13:27:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:27:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-04-13T13:27:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:27:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-04-13T13:27:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:27:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-04-13T13:27:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_13': 2024-04-13T13:27:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:27:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:27:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12ed4f20c2b4 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12ed4f20c2b4 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:15 2024 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:15 2024 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:15 2024 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: bf426828-2e69-4ba4-a59f-e3526fdc760d 2024-04-13T13:27:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:27:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:27:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_13 2024-04-13T13:27:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:27:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:27:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=11 2024-04-13T13:27:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:27:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:27:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 6256640 bytes 2024-04-13T13:27:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 43849728 bytes 2024-04-13T13:27:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 57192448 bytes 2024-04-13T13:27:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 105296896 bytes 2024-04-13T13:27:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 126904320 bytes 2024-04-13T13:27:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 142655488 bytes 2024-04-13T13:27:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 225583104 bytes 2024-04-13T13:27:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 239273984 bytes 2024-04-13T13:27:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 342777856 bytes 2024-04-13T13:27:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 368205824 bytes 2024-04-13T13:27:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 399206400 bytes 2024-04-13T13:27:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 451766272 bytes 2024-04-13T13:27:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 461453312 bytes 2024-04-13T13:27:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 510724096 bytes 2024-04-13T13:27:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 553425920 bytes 2024-04-13T13:27:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 572240896 bytes 2024-04-13T13:27:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 617289728 bytes 2024-04-13T13:27:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 629896192 bytes 2024-04-13T13:27:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 630508544 bytes 2024-04-13T13:27:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 666401792 bytes 2024-04-13T13:27:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 718538752 bytes 2024-04-13T13:27:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 774509568 bytes 2024-04-13T13:27:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 823891968 bytes 2024-04-13T13:27:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 877497344 bytes 2024-04-13T13:27:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 931791872 bytes 2024-04-13T13:27:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 935706624 bytes 2024-04-13T13:27:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 957970432 bytes 2024-04-13T13:27:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1023260672 bytes 2024-04-13T13:27:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1039508480 bytes 2024-04-13T13:27:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1082327040 bytes 2024-04-13T13:27:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1118754816 bytes 2024-04-13T13:27:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1203449856 bytes 2024-04-13T13:27:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1237693440 bytes 2024-04-13T13:27:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1264872448 bytes 2024-04-13T13:27:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1288233984 bytes 2024-04-13T13:27:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1311131648 bytes 2024-04-13T13:27:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1313036288 bytes 2024-04-13T13:27:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1326259200 bytes 2024-04-13T13:27:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1374585856 bytes 2024-04-13T13:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1375197184 bytes 2024-04-13T13:27:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1440215040 bytes 2024-04-13T13:27:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1511995392 bytes 2024-04-13T13:27:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1564183552 bytes 2024-04-13T13:27:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1615038464 bytes 2024-04-13T13:27:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1618963456 bytes 2024-04-13T13:27:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1654184960 bytes 2024-04-13T13:27:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1662467072 bytes 2024-04-13T13:27:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1693629440 bytes 2024-04-13T13:27:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1762863104 bytes 2024-04-13T13:27:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1834799104 bytes 2024-04-13T13:27:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1890784256 bytes 2024-04-13T13:27:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1941840896 bytes 2024-04-13T13:27:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2012579840 bytes 2024-04-13T13:27:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2048943104 bytes 2024-04-13T13:27:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2103375872 bytes 2024-04-13T13:27:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2161623040 bytes 2024-04-13T13:27:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2177950720 bytes 2024-04-13T13:27:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2263958528 bytes 2024-04-13T13:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2267194368 bytes 2024-04-13T13:27:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2306008064 bytes 2024-04-13T13:27:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2336362496 bytes 2024-04-13T13:27:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2343550976 bytes 2024-04-13T13:27:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2402686976 bytes 2024-04-13T13:27:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2474509312 bytes 2024-04-13T13:27:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2555060224 bytes 2024-04-13T13:27:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2569976832 bytes 2024-04-13T13:27:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2592943104 bytes 2024-04-13T13:27:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2606199808 bytes 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_14 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_14 128M 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:27:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:27:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:27:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2024-04-13T13:27:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:27:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-04-13T13:27:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:27:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-04-13T13:27:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:27:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_14': 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12f88c7957a2 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12f88c7957a2 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:27:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:27:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:27:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:27 2024 2024-04-13T13:27:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:27 2024 2024-04-13T13:27:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:27 2024 2024-04-13T13:27:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:27:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:27:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: dc2077e0-ec0a-4427-ab08-fc9e2fa06c71 2024-04-13T13:27:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:27:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:27:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_14 2024-04-13T13:27:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:27:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:27:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=11 2024-04-13T13:27:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:27:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:27:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 42477568 bytes 2024-04-13T13:27:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 88317952 bytes 2024-04-13T13:27:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 128416768 bytes 2024-04-13T13:27:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 181702656 bytes 2024-04-13T13:27:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 214501376 bytes 2024-04-13T13:27:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 281861120 bytes 2024-04-13T13:27:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 286063616 bytes 2024-04-13T13:27:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 338201600 bytes 2024-04-13T13:27:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 378198016 bytes 2024-04-13T13:27:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 447261696 bytes 2024-04-13T13:27:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 516131840 bytes 2024-04-13T13:27:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 539161600 bytes 2024-04-13T13:27:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 587033600 bytes 2024-04-13T13:27:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 615689216 bytes 2024-04-13T13:27:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 681542656 bytes 2024-04-13T13:27:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 723998720 bytes 2024-04-13T13:27:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 742339584 bytes 2024-04-13T13:27:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 781786112 bytes 2024-04-13T13:27:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 790904832 bytes 2024-04-13T13:27:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 855229440 bytes 2024-04-13T13:27:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 865744896 bytes 2024-04-13T13:27:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 931005440 bytes 2024-04-13T13:27:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 962961408 bytes 2024-04-13T13:27:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 989837312 bytes 2024-04-13T13:27:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1059197952 bytes 2024-04-13T13:27:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1119738880 bytes 2024-04-13T13:27:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1143306240 bytes 2024-04-13T13:27:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1167682560 bytes 2024-04-13T13:27:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1212023808 bytes 2024-04-13T13:27:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1225269248 bytes 2024-04-13T13:27:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1252493312 bytes 2024-04-13T13:27:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1301641216 bytes 2024-04-13T13:27:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1305412608 bytes 2024-04-13T13:27:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1321903104 bytes 2024-04-13T13:27:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1336585216 bytes 2024-04-13T13:27:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1400420352 bytes 2024-04-13T13:27:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1437606912 bytes 2024-04-13T13:27:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1466748928 bytes 2024-04-13T13:27:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1573997568 bytes 2024-04-13T13:27:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1640989696 bytes 2024-04-13T13:27:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1645161472 bytes 2024-04-13T13:27:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1697137664 bytes 2024-04-13T13:27:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1734372352 bytes 2024-04-13T13:27:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1734985728 bytes 2024-04-13T13:27:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1760132096 bytes 2024-04-13T13:27:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1817934848 bytes 2024-04-13T13:27:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1882541056 bytes 2024-04-13T13:27:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1896851456 bytes 2024-04-13T13:27:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1959716864 bytes 2024-04-13T13:27:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1997253632 bytes 2024-04-13T13:27:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2041890816 bytes 2024-04-13T13:27:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2047363072 bytes 2024-04-13T13:27:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2109409280 bytes 2024-04-13T13:27:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2155421696 bytes 2024-04-13T13:27:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2189928448 bytes 2024-04-13T13:27:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2218692608 bytes 2024-04-13T13:27:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2248928256 bytes 2024-04-13T13:27:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2285665280 bytes 2024-04-13T13:27:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2386336768 bytes 2024-04-13T13:27:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2412180480 bytes 2024-04-13T13:27:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2486898688 bytes 2024-04-13T13:27:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2559088640 bytes 2024-04-13T13:27:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2646505472 bytes 2024-04-13T13:27:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2675624960 bytes 2024-04-13T13:27:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2696522752 bytes 2024-04-13T13:27:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_15 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_15 128M 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:27:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:27:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2024-04-13T13:27:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:27:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-04-13T13:27:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:27:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-04-13T13:27:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:27:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_15': 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13018ba0af95 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13018ba0af95 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:39 2024 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:39 2024 2024-04-13T13:27:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:39 2024 2024-04-13T13:27:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:27:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:27:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: f38ae892-421e-4484-b8f6-de0cb55a7833 2024-04-13T13:27:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:27:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:27:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_15 2024-04-13T13:27:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:27:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:27:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=6 2024-04-13T13:27:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:27:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:27:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 22404096 bytes 2024-04-13T13:27:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 99416064 bytes 2024-04-13T13:27:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 189392896 bytes 2024-04-13T13:27:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 236806144 bytes 2024-04-13T13:27:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 278840320 bytes 2024-04-13T13:27:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 299703296 bytes 2024-04-13T13:27:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 332096512 bytes 2024-04-13T13:27:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 356302848 bytes 2024-04-13T13:27:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 372506624 bytes 2024-04-13T13:27:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 397072384 bytes 2024-04-13T13:27:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 437937152 bytes 2024-04-13T13:27:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 461117440 bytes 2024-04-13T13:27:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 489626624 bytes 2024-04-13T13:27:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 562937856 bytes 2024-04-13T13:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 635681792 bytes 2024-04-13T13:27:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 676790272 bytes 2024-04-13T13:27:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 708442112 bytes 2024-04-13T13:27:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 775014400 bytes 2024-04-13T13:27:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 778428416 bytes 2024-04-13T13:27:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 804251648 bytes 2024-04-13T13:27:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 839859200 bytes 2024-04-13T13:27:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 863075328 bytes 2024-04-13T13:27:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 882685952 bytes 2024-04-13T13:27:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 906817536 bytes 2024-04-13T13:27:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 967802880 bytes 2024-04-13T13:27:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1007047680 bytes 2024-04-13T13:27:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1066609664 bytes 2024-04-13T13:27:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1090383872 bytes 2024-04-13T13:27:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1136651264 bytes 2024-04-13T13:27:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1170103296 bytes 2024-04-13T13:27:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1225909248 bytes 2024-04-13T13:27:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1266417664 bytes 2024-04-13T13:27:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1324620800 bytes 2024-04-13T13:27:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1416437760 bytes 2024-04-13T13:27:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1432446976 bytes 2024-04-13T13:27:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:27:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_16 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:27:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_16 128M 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2024-04-13T13:27:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:27:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-04-13T13:27:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:27:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-04-13T13:27:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:27:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-04-13T13:27:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_16': 2024-04-13T13:27:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 130a685ccf7e 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.130a685ccf7e 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:46 2024 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:46 2024 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:46 2024 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:27:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 53de06e5-2f50-4532-ba7a-3aa9dd41bcfd 2024-04-13T13:27:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:27:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:27:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_16 2024-04-13T13:27:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:27:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:27:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=9 2024-04-13T13:27:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:27:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:27:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 48689152 bytes 2024-04-13T13:27:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 106418176 bytes 2024-04-13T13:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 133605376 bytes 2024-04-13T13:27:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 183957504 bytes 2024-04-13T13:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 233306112 bytes 2024-04-13T13:27:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 282351616 bytes 2024-04-13T13:27:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 313176064 bytes 2024-04-13T13:27:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 398434304 bytes 2024-04-13T13:27:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 414212096 bytes 2024-04-13T13:27:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 439401472 bytes 2024-04-13T13:27:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 511108096 bytes 2024-04-13T13:27:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 583397376 bytes 2024-04-13T13:27:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 616037376 bytes 2024-04-13T13:27:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 710380544 bytes 2024-04-13T13:27:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 791829504 bytes 2024-04-13T13:27:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 825894912 bytes 2024-04-13T13:27:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 890189824 bytes 2024-04-13T13:27:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 934194176 bytes 2024-04-13T13:27:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 992392192 bytes 2024-04-13T13:27:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 1074907136 bytes 2024-04-13T13:27:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 1114797056 bytes 2024-04-13T13:27:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1164670976 bytes 2024-04-13T13:27:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1193466880 bytes 2024-04-13T13:27:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1283995648 bytes 2024-04-13T13:27:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1288946688 bytes 2024-04-13T13:27:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1349478400 bytes 2024-04-13T13:27:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1412344832 bytes 2024-04-13T13:27:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1442423808 bytes 2024-04-13T13:27:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1451066368 bytes 2024-04-13T13:27:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1545308160 bytes 2024-04-13T13:27:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1594021888 bytes 2024-04-13T13:27:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1606882304 bytes 2024-04-13T13:27:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1642424320 bytes 2024-04-13T13:27:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1674816512 bytes 2024-04-13T13:27:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1701267456 bytes 2024-04-13T13:27:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1764246528 bytes 2024-04-13T13:27:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1772182528 bytes 2024-04-13T13:27:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1812121600 bytes 2024-04-13T13:27:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1882293248 bytes 2024-04-13T13:27:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1933519872 bytes 2024-04-13T13:27:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1934137344 bytes 2024-04-13T13:27:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1972192256 bytes 2024-04-13T13:27:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 2028357632 bytes 2024-04-13T13:27:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 2064061440 bytes 2024-04-13T13:27:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 2064668672 bytes 2024-04-13T13:27:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2084238336 bytes 2024-04-13T13:27:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2154375168 bytes 2024-04-13T13:27:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2198106112 bytes 2024-04-13T13:27:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2242981888 bytes 2024-04-13T13:27:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2273199104 bytes 2024-04-13T13:27:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2331214848 bytes 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_17 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_17 128M 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:27:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:27:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:27:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:27:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:27:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2024-04-13T13:27:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:27:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-04-13T13:27:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:27:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:27:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-04-13T13:27:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:27:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-04-13T13:27:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_17': 2024-04-13T13:27:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:27:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:27:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:27:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1312a9c68acb 2024-04-13T13:27:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1312a9c68acb 2024-04-13T13:27:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:56 2024 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:56 2024 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:56 2024 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: d2b64f41-9de5-4eab-8f44-349912e93682 2024-04-13T13:27:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:27:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:27:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_17 2024-04-13T13:27:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:27:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:27:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:27:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:27:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=11 2024-04-13T13:27:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:27:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:27:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 37818368 bytes 2024-04-13T13:27:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 66942976 bytes 2024-04-13T13:27:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 106955776 bytes 2024-04-13T13:27:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 154260480 bytes 2024-04-13T13:27:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 170438656 bytes 2024-04-13T13:27:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 202169344 bytes 2024-04-13T13:27:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 210984960 bytes 2024-04-13T13:27:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 281023488 bytes 2024-04-13T13:27:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 333421568 bytes 2024-04-13T13:27:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 357692416 bytes 2024-04-13T13:27:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 388764672 bytes 2024-04-13T13:27:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 476324864 bytes 2024-04-13T13:28:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 523275264 bytes 2024-04-13T13:28:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 536454144 bytes 2024-04-13T13:28:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 580063232 bytes 2024-04-13T13:28:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 587826176 bytes 2024-04-13T13:28:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 660528128 bytes 2024-04-13T13:28:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 709641216 bytes 2024-04-13T13:28:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 718799872 bytes 2024-04-13T13:28:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 719396864 bytes 2024-04-13T13:28:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 800124928 bytes 2024-04-13T13:28:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 828439552 bytes 2024-04-13T13:28:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 844311552 bytes 2024-04-13T13:28:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 876588032 bytes 2024-04-13T13:28:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 917853184 bytes 2024-04-13T13:28:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 920718336 bytes 2024-04-13T13:28:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 986309632 bytes 2024-04-13T13:28:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1000404992 bytes 2024-04-13T13:28:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1061246976 bytes 2024-04-13T13:28:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1113388032 bytes 2024-04-13T13:28:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1182687232 bytes 2024-04-13T13:28:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1285324800 bytes 2024-04-13T13:28:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1323811840 bytes 2024-04-13T13:28:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1369230336 bytes 2024-04-13T13:28:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1414182912 bytes 2024-04-13T13:28:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1469724672 bytes 2024-04-13T13:28:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1560972288 bytes 2024-04-13T13:28:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1639562240 bytes 2024-04-13T13:28:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1668691968 bytes 2024-04-13T13:28:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1701382144 bytes 2024-04-13T13:28:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1730801664 bytes 2024-04-13T13:28:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1771756544 bytes 2024-04-13T13:28:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1799704576 bytes 2024-04-13T13:28:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1871360000 bytes 2024-04-13T13:28:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1920214016 bytes 2024-04-13T13:28:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1920827392 bytes 2024-04-13T13:28:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2004466688 bytes 2024-04-13T13:28:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2083233792 bytes 2024-04-13T13:28:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2153450496 bytes 2024-04-13T13:28:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2191909888 bytes 2024-04-13T13:28:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2227517440 bytes 2024-04-13T13:28:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2256386048 bytes 2024-04-13T13:28:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2303076352 bytes 2024-04-13T13:28:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2378163200 bytes 2024-04-13T13:28:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2449319936 bytes 2024-04-13T13:28:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2485207040 bytes 2024-04-13T13:28:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2522989568 bytes 2024-04-13T13:28:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2576115712 bytes 2024-04-13T13:28:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2602883072 bytes 2024-04-13T13:28:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2666091520 bytes 2024-04-13T13:28:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2732318720 bytes 2024-04-13T13:28:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2767988736 bytes 2024-04-13T13:28:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2778769408 bytes 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_18 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_18 128M 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2024-04-13T13:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-04-13T13:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-04-13T13:28:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:28:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_18': 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 131af9e6f6fd 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.131af9e6f6fd 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:08 2024 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:08 2024 2024-04-13T13:28:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:08 2024 2024-04-13T13:28:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:28:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:28:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 807d8287-a59d-4634-8f59-0370463ec872 2024-04-13T13:28:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_18 2024-04-13T13:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=6 2024-04-13T13:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:28:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 11578368 bytes 2024-04-13T13:28:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 64062464 bytes 2024-04-13T13:28:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 82178048 bytes 2024-04-13T13:28:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 107505664 bytes 2024-04-13T13:28:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 122626048 bytes 2024-04-13T13:28:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 190115840 bytes 2024-04-13T13:28:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 211313664 bytes 2024-04-13T13:28:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 258983936 bytes 2024-04-13T13:28:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 306078720 bytes 2024-04-13T13:28:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 334769152 bytes 2024-04-13T13:28:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 414026752 bytes 2024-04-13T13:28:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 434956288 bytes 2024-04-13T13:28:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 440563712 bytes 2024-04-13T13:28:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 460807168 bytes 2024-04-13T13:28:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 548097024 bytes 2024-04-13T13:28:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 581615616 bytes 2024-04-13T13:28:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 646962176 bytes 2024-04-13T13:28:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 670320640 bytes 2024-04-13T13:28:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 684366848 bytes 2024-04-13T13:28:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 687778816 bytes 2024-04-13T13:28:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 688384000 bytes 2024-04-13T13:28:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 740162560 bytes 2024-04-13T13:28:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 740787200 bytes 2024-04-13T13:28:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 811577344 bytes 2024-04-13T13:28:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 835648512 bytes 2024-04-13T13:28:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 869345280 bytes 2024-04-13T13:28:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 901951488 bytes 2024-04-13T13:28:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 979520512 bytes 2024-04-13T13:28:14.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1024785408 bytes 2024-04-13T13:28:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1030658048 bytes 2024-04-13T13:28:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1052350464 bytes 2024-04-13T13:28:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1100054528 bytes 2024-04-13T13:28:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1148820480 bytes 2024-04-13T13:28:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1186722816 bytes 2024-04-13T13:28:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1238782976 bytes 2024-04-13T13:28:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1254618112 bytes 2024-04-13T13:28:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1285590016 bytes 2024-04-13T13:28:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1308752896 bytes 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_19 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_19 128M 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2024-04-13T13:28:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:28:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-04-13T13:28:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:28:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-04-13T13:28:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:28:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-04-13T13:28:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_19': 2024-04-13T13:28:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:28:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1325605ffc77 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1325605ffc77 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:16 2024 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:16 2024 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:16 2024 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 0c18f79e-f883-47a8-84ef-a7789e73b4ba 2024-04-13T13:28:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:28:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:28:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_19 2024-04-13T13:28:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:28:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:28:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=8 2024-04-13T13:28:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:28:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:28:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 5950464 bytes 2024-04-13T13:28:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 73175040 bytes 2024-04-13T13:28:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 114030592 bytes 2024-04-13T13:28:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 162620416 bytes 2024-04-13T13:28:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 221877248 bytes 2024-04-13T13:28:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 239686656 bytes 2024-04-13T13:28:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 256167936 bytes 2024-04-13T13:28:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 292792320 bytes 2024-04-13T13:28:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 347487232 bytes 2024-04-13T13:28:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 420995072 bytes 2024-04-13T13:28:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 479756288 bytes 2024-04-13T13:28:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 494794752 bytes 2024-04-13T13:28:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 524371968 bytes 2024-04-13T13:28:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 582969344 bytes 2024-04-13T13:28:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 613201920 bytes 2024-04-13T13:28:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 635227136 bytes 2024-04-13T13:28:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 691100672 bytes 2024-04-13T13:28:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 734865408 bytes 2024-04-13T13:28:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 778930176 bytes 2024-04-13T13:28:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 821794816 bytes 2024-04-13T13:28:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 885453824 bytes 2024-04-13T13:28:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 898626560 bytes 2024-04-13T13:28:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1003300864 bytes 2024-04-13T13:28:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1060311040 bytes 2024-04-13T13:28:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1101329408 bytes 2024-04-13T13:28:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1170231296 bytes 2024-04-13T13:28:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1211372544 bytes 2024-04-13T13:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1243206656 bytes 2024-04-13T13:28:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1300893696 bytes 2024-04-13T13:28:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1334657024 bytes 2024-04-13T13:28:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1342005248 bytes 2024-04-13T13:28:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1399989248 bytes 2024-04-13T13:28:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1411420160 bytes 2024-04-13T13:28:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1421654016 bytes 2024-04-13T13:28:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1530028032 bytes 2024-04-13T13:28:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1568602112 bytes 2024-04-13T13:28:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1607787520 bytes 2024-04-13T13:28:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1639561216 bytes 2024-04-13T13:28:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1660795904 bytes 2024-04-13T13:28:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1702470656 bytes 2024-04-13T13:28:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1727237120 bytes 2024-04-13T13:28:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1767490560 bytes 2024-04-13T13:28:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1812903936 bytes 2024-04-13T13:28:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1894507520 bytes 2024-04-13T13:28:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1938761728 bytes 2024-04-13T13:28:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1939380224 bytes 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_20 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_20 128M 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2024-04-13T13:28:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:28:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-04-13T13:28:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:28:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-04-13T13:28:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:28:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_20': 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 132ed3a30316 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.132ed3a30316 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:28:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:28:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:25 2024 2024-04-13T13:28:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:25 2024 2024-04-13T13:28:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:25 2024 2024-04-13T13:28:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:28:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:28:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: c3ae73db-b27b-4d07-9b28-0323537a736a 2024-04-13T13:28:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:28:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:28:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_20 2024-04-13T13:28:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:28:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:28:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=5 2024-04-13T13:28:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:28:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:28:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 7133184 bytes 2024-04-13T13:28:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 54502400 bytes 2024-04-13T13:28:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 110137344 bytes 2024-04-13T13:28:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 119271424 bytes 2024-04-13T13:28:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 132703232 bytes 2024-04-13T13:28:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 133326848 bytes 2024-04-13T13:28:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 150126592 bytes 2024-04-13T13:28:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 192126976 bytes 2024-04-13T13:28:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 206816256 bytes 2024-04-13T13:28:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 242835456 bytes 2024-04-13T13:28:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 282851328 bytes 2024-04-13T13:28:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 338992128 bytes 2024-04-13T13:28:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 421898240 bytes 2024-04-13T13:28:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 468046848 bytes 2024-04-13T13:28:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 471106560 bytes 2024-04-13T13:28:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 531622912 bytes 2024-04-13T13:28:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 556772352 bytes 2024-04-13T13:28:28.814 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6315121005087685) > weight (0.25) 2024-04-13T13:28:28.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2507535246255931) > weight (0.25) 2024-04-13T13:28:28.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9033492413189331) > weight (0.25) 2024-04-13T13:28:28.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9274490879065953) > weight (0.25) 2024-04-13T13:28:28.815 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-04-13T13:28:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 599777280 bytes 2024-04-13T13:28:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 648218624 bytes 2024-04-13T13:28:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 719267840 bytes 2024-04-13T13:28:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 758998016 bytes 2024-04-13T13:28:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 819600384 bytes 2024-04-13T13:28:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 844773376 bytes 2024-04-13T13:28:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 845381632 bytes 2024-04-13T13:28:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 881894400 bytes 2024-04-13T13:28:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 921330688 bytes 2024-04-13T13:28:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 982572032 bytes 2024-04-13T13:28:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1080154112 bytes 2024-04-13T13:28:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1151157248 bytes 2024-04-13T13:28:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1193150464 bytes 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_21 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_21 128M 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:28:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2024-04-13T13:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-04-13T13:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-04-13T13:28:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:28:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-04-13T13:28:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_21': 2024-04-13T13:28:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:28:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:28:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:28:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1337694b5989 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1337694b5989 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:31 2024 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:31 2024 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:31 2024 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 213e24cb-89ef-475f-84d5-92800a3d1efb 2024-04-13T13:28:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:28:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:28:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_21 2024-04-13T13:28:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:28:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:28:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=11 2024-04-13T13:28:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:28:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:28:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 36147200 bytes 2024-04-13T13:28:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 150718464 bytes 2024-04-13T13:28:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 181896192 bytes 2024-04-13T13:28:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 198514688 bytes 2024-04-13T13:28:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 271042560 bytes 2024-04-13T13:28:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 332936192 bytes 2024-04-13T13:28:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 416447488 bytes 2024-04-13T13:28:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 440093696 bytes 2024-04-13T13:28:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 478116864 bytes 2024-04-13T13:28:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 525787136 bytes 2024-04-13T13:28:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 591252480 bytes 2024-04-13T13:28:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 613079040 bytes 2024-04-13T13:28:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 658692096 bytes 2024-04-13T13:28:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 696478720 bytes 2024-04-13T13:28:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 713639936 bytes 2024-04-13T13:28:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 844514304 bytes 2024-04-13T13:28:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 864844800 bytes 2024-04-13T13:28:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 903217152 bytes 2024-04-13T13:28:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 939614208 bytes 2024-04-13T13:28:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 949143552 bytes 2024-04-13T13:28:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 969828352 bytes 2024-04-13T13:28:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1022711808 bytes 2024-04-13T13:28:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1085937664 bytes 2024-04-13T13:28:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1162625024 bytes 2024-04-13T13:28:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1211888640 bytes 2024-04-13T13:28:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1229192192 bytes 2024-04-13T13:28:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1233510400 bytes 2024-04-13T13:28:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1276326912 bytes 2024-04-13T13:28:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1316020224 bytes 2024-04-13T13:28:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1373927424 bytes 2024-04-13T13:28:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1399477248 bytes 2024-04-13T13:28:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1428826112 bytes 2024-04-13T13:28:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1458401280 bytes 2024-04-13T13:28:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1477458944 bytes 2024-04-13T13:28:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1500745728 bytes 2024-04-13T13:28:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1551943680 bytes 2024-04-13T13:28:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1575817216 bytes 2024-04-13T13:28:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1594754048 bytes 2024-04-13T13:28:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1659220992 bytes 2024-04-13T13:28:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1667468288 bytes 2024-04-13T13:28:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1684064256 bytes 2024-04-13T13:28:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1734246400 bytes 2024-04-13T13:28:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1817880576 bytes 2024-04-13T13:28:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1838534656 bytes 2024-04-13T13:28:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1907800064 bytes 2024-04-13T13:28:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1930730496 bytes 2024-04-13T13:28:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1964812288 bytes 2024-04-13T13:28:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2019076096 bytes 2024-04-13T13:28:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2068744192 bytes 2024-04-13T13:28:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2069363712 bytes 2024-04-13T13:28:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2100112384 bytes 2024-04-13T13:28:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2171119616 bytes 2024-04-13T13:28:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2194596864 bytes 2024-04-13T13:28:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2200189952 bytes 2024-04-13T13:28:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2200800256 bytes 2024-04-13T13:28:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2236386304 bytes 2024-04-13T13:28:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2285987840 bytes 2024-04-13T13:28:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2330997760 bytes 2024-04-13T13:28:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2361318400 bytes 2024-04-13T13:28:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2401971200 bytes 2024-04-13T13:28:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2409847808 bytes 2024-04-13T13:28:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2484368384 bytes 2024-04-13T13:28:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2508770304 bytes 2024-04-13T13:28:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2533602304 bytes 2024-04-13T13:28:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2602913792 bytes 2024-04-13T13:28:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:28:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:28:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_22 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_22 128M 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2024-04-13T13:28:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:28:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-04-13T13:28:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:28:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-04-13T13:28:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:28:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-04-13T13:28:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_22': 2024-04-13T13:28:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:28:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:28:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:28:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 134169b234be 2024-04-13T13:28:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.134169b234be 2024-04-13T13:28:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:28:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:42 2024 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:42 2024 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:42 2024 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: e58af553-340d-46ba-a87f-856f475da231 2024-04-13T13:28:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:28:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:28:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_22 2024-04-13T13:28:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:28:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:28:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=6 2024-04-13T13:28:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:28:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:28:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 33201152 bytes 2024-04-13T13:28:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 77144064 bytes 2024-04-13T13:28:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 98733056 bytes 2024-04-13T13:28:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 127127552 bytes 2024-04-13T13:28:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 190547968 bytes 2024-04-13T13:28:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 220595200 bytes 2024-04-13T13:28:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 295625728 bytes 2024-04-13T13:28:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 316239872 bytes 2024-04-13T13:28:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 319917056 bytes 2024-04-13T13:28:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 339078144 bytes 2024-04-13T13:28:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 382652416 bytes 2024-04-13T13:28:45.712 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6351886418036602) > weight (0.25) 2024-04-13T13:28:45.712 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.884010890003608) > weight (0.25) 2024-04-13T13:28:45.712 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-13T13:28:45.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-13T13:28:45.713 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-13T13:28:45.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi122.stderr:2024-04-13T13:28:45.719+0000 7fcf22ff9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 43994) UID: 1000 2024-04-13T13:28:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 398758912 bytes 2024-04-13T13:28:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 482192384 bytes 2024-04-13T13:28:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 526969856 bytes 2024-04-13T13:28:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 543380480 bytes 2024-04-13T13:28:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 629576704 bytes 2024-04-13T13:28:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 651407360 bytes 2024-04-13T13:28:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 690742272 bytes 2024-04-13T13:28:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 714852352 bytes 2024-04-13T13:28:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 798098432 bytes 2024-04-13T13:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 867372032 bytes 2024-04-13T13:28:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 878657536 bytes 2024-04-13T13:28:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 921192448 bytes 2024-04-13T13:28:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 945921024 bytes 2024-04-13T13:28:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 991275008 bytes 2024-04-13T13:28:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1030379520 bytes 2024-04-13T13:28:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1031694336 bytes 2024-04-13T13:28:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1046576128 bytes 2024-04-13T13:28:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1048667136 bytes 2024-04-13T13:28:48.713 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-13T13:28:48.714 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:28:48.714 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-13T13:28:48.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-13T13:28:48.714 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-13T13:28:48.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-13T13:28:48.717 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-04-13T13:28:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1077773312 bytes 2024-04-13T13:28:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1088464896 bytes 2024-04-13T13:28:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1141567488 bytes 2024-04-13T13:28:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1193768960 bytes 2024-04-13T13:28:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1207318528 bytes 2024-04-13T13:28:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1247617024 bytes 2024-04-13T13:28:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1257263104 bytes 2024-04-13T13:28:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:28:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:28:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:28:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:28:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:28:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_23 2024-04-13T13:28:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-04-13T13:28:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_23 128M 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:28:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:28:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:28:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:28:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2024-04-13T13:28:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:28:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-04-13T13:28:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:28:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-04-13T13:28:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:28:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-04-13T13:28:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_23': 2024-04-13T13:28:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:28:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:28:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:28:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1353222a0946 2024-04-13T13:28:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1353222a0946 2024-04-13T13:28:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:28:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:49 2024 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:49 2024 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:49 2024 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8c509f7e-2bfc-4798-88c9-e8a036e7a227 2024-04-13T13:28:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:28:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:28:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_23 2024-04-13T13:28:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:28:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:28:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=8 2024-04-13T13:28:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:28:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:28:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 37432320 bytes 2024-04-13T13:28:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 78761984 bytes 2024-04-13T13:28:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 86934528 bytes 2024-04-13T13:28:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 124049408 bytes 2024-04-13T13:28:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 204101632 bytes 2024-04-13T13:28:51.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 221118464 bytes 2024-04-13T13:28:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 226721792 bytes 2024-04-13T13:28:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 349588480 bytes 2024-04-13T13:28:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 382280704 bytes 2024-04-13T13:28:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 382878720 bytes 2024-04-13T13:28:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 408804352 bytes 2024-04-13T13:28:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 427208704 bytes 2024-04-13T13:28:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 430236672 bytes 2024-04-13T13:28:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 506592256 bytes 2024-04-13T13:28:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 522381312 bytes 2024-04-13T13:28:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 566169600 bytes 2024-04-13T13:28:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 574845952 bytes 2024-04-13T13:28:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 647461888 bytes 2024-04-13T13:28:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 662093824 bytes 2024-04-13T13:28:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 728519680 bytes 2024-04-13T13:28:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 747537408 bytes 2024-04-13T13:28:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 752444416 bytes 2024-04-13T13:28:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 798328832 bytes 2024-04-13T13:28:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 805841920 bytes 2024-04-13T13:28:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 837608448 bytes 2024-04-13T13:28:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 838183936 bytes 2024-04-13T13:28:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 871272448 bytes 2024-04-13T13:28:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 934602752 bytes 2024-04-13T13:28:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 966735872 bytes 2024-04-13T13:28:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1036795904 bytes 2024-04-13T13:28:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1111398400 bytes 2024-04-13T13:28:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1141625856 bytes 2024-04-13T13:28:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1163119616 bytes 2024-04-13T13:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1183272960 bytes 2024-04-13T13:28:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1224392704 bytes 2024-04-13T13:28:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1291350016 bytes 2024-04-13T13:28:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1296001024 bytes 2024-04-13T13:28:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1313979392 bytes 2024-04-13T13:28:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1340974080 bytes 2024-04-13T13:28:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1375280128 bytes 2024-04-13T13:28:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1444589568 bytes 2024-04-13T13:28:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1468019712 bytes 2024-04-13T13:28:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1492504576 bytes 2024-04-13T13:28:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1537280000 bytes 2024-04-13T13:28:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1553966080 bytes 2024-04-13T13:28:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1592854528 bytes 2024-04-13T13:28:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1647084544 bytes 2024-04-13T13:28:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1691340800 bytes 2024-04-13T13:28:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1733770240 bytes 2024-04-13T13:28:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1753121792 bytes 2024-04-13T13:28:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:28:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:28:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:28:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:28:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_24 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_24 128M 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:28:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2024-04-13T13:28:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:28:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-04-13T13:28:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:28:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:28:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-04-13T13:28:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:28:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_24': 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 135c718a791 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.135c718a791 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:28:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:28:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:58 2024 2024-04-13T13:28:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:58 2024 2024-04-13T13:28:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:58 2024 2024-04-13T13:28:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:28:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:28:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 91c5558e-79d7-4410-a35b-b56664447d57 2024-04-13T13:28:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:28:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:28:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_24 2024-04-13T13:28:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:28:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:28:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:28:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:28:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=5 2024-04-13T13:28:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:28:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:29:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 41835520 bytes 2024-04-13T13:29:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 79154176 bytes 2024-04-13T13:29:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 85255168 bytes 2024-04-13T13:29:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 129153024 bytes 2024-04-13T13:29:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 181038080 bytes 2024-04-13T13:29:00.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 213675008 bytes 2024-04-13T13:29:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 246921216 bytes 2024-04-13T13:29:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 255323136 bytes 2024-04-13T13:29:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 310198272 bytes 2024-04-13T13:29:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 377038848 bytes 2024-04-13T13:29:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 412489728 bytes 2024-04-13T13:29:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 438462464 bytes 2024-04-13T13:29:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 513361920 bytes 2024-04-13T13:29:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 541286400 bytes 2024-04-13T13:29:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 577212416 bytes 2024-04-13T13:29:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 646286336 bytes 2024-04-13T13:29:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 698586112 bytes 2024-04-13T13:29:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 746046464 bytes 2024-04-13T13:29:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 754027520 bytes 2024-04-13T13:29:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 812575744 bytes 2024-04-13T13:29:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 874078208 bytes 2024-04-13T13:29:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 902310912 bytes 2024-04-13T13:29:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1019792384 bytes 2024-04-13T13:29:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1077863424 bytes 2024-04-13T13:29:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1120735232 bytes 2024-04-13T13:29:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1190598656 bytes 2024-04-13T13:29:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1238219776 bytes 2024-04-13T13:29:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1256005632 bytes 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_25 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_25 128M 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:29:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:29:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2024-04-13T13:29:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:29:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-04-13T13:29:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:29:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-04-13T13:29:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:29:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-04-13T13:29:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_25': 2024-04-13T13:29:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1364ed018da3 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1364ed018da3 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:04 2024 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:04 2024 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:04 2024 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 9c3874e8-22df-4fdb-a27a-7e2740eed944 2024-04-13T13:29:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:29:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:29:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_25 2024-04-13T13:29:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:29:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:29:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=6 2024-04-13T13:29:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:29:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:29:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 76410880 bytes 2024-04-13T13:29:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 79110144 bytes 2024-04-13T13:29:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 180554752 bytes 2024-04-13T13:29:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 232245248 bytes 2024-04-13T13:29:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 268814336 bytes 2024-04-13T13:29:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 299537408 bytes 2024-04-13T13:29:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 392844288 bytes 2024-04-13T13:29:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 418546688 bytes 2024-04-13T13:29:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 452602880 bytes 2024-04-13T13:29:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 469722112 bytes 2024-04-13T13:29:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 538081280 bytes 2024-04-13T13:29:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 582841344 bytes 2024-04-13T13:29:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 628844544 bytes 2024-04-13T13:29:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 644271104 bytes 2024-04-13T13:29:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 644885504 bytes 2024-04-13T13:29:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 751003648 bytes 2024-04-13T13:29:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 769571840 bytes 2024-04-13T13:29:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 780888064 bytes 2024-04-13T13:29:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 807166976 bytes 2024-04-13T13:29:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 863936512 bytes 2024-04-13T13:29:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 878320640 bytes 2024-04-13T13:29:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 944417792 bytes 2024-04-13T13:29:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 981368832 bytes 2024-04-13T13:29:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1001193472 bytes 2024-04-13T13:29:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1060009984 bytes 2024-04-13T13:29:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1085169664 bytes 2024-04-13T13:29:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1111906304 bytes 2024-04-13T13:29:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1156179968 bytes 2024-04-13T13:29:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1183816704 bytes 2024-04-13T13:29:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1209000960 bytes 2024-04-13T13:29:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1261801472 bytes 2024-04-13T13:29:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1318450176 bytes 2024-04-13T13:29:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1354651648 bytes 2024-04-13T13:29:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1395519488 bytes 2024-04-13T13:29:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1421914112 bytes 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_26 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_26 128M 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2024-04-13T13:29:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-04-13T13:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-04-13T13:29:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:29:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-04-13T13:29:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_26': 2024-04-13T13:29:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:29:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:29:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:29:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 136d1c158451 2024-04-13T13:29:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.136d1c158451 2024-04-13T13:29:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:11 2024 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:11 2024 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:11 2024 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: cc11b282-5f39-432a-9e34-fd7d24025087 2024-04-13T13:29:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:29:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:29:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_26 2024-04-13T13:29:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:29:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:29:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=5 2024-04-13T13:29:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:29:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:29:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 54506496 bytes 2024-04-13T13:29:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 137634816 bytes 2024-04-13T13:29:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 166432768 bytes 2024-04-13T13:29:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 214272000 bytes 2024-04-13T13:29:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 299795456 bytes 2024-04-13T13:29:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 309696512 bytes 2024-04-13T13:29:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 355574784 bytes 2024-04-13T13:29:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 387604480 bytes 2024-04-13T13:29:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 470736896 bytes 2024-04-13T13:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 471359488 bytes 2024-04-13T13:29:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 518348800 bytes 2024-04-13T13:29:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 541260800 bytes 2024-04-13T13:29:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 575613952 bytes 2024-04-13T13:29:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 612835328 bytes 2024-04-13T13:29:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 660205568 bytes 2024-04-13T13:29:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 760766464 bytes 2024-04-13T13:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 811589632 bytes 2024-04-13T13:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 876314624 bytes 2024-04-13T13:29:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 919056384 bytes 2024-04-13T13:29:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 951115776 bytes 2024-04-13T13:29:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 1000446976 bytes 2024-04-13T13:29:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1001080832 bytes 2024-04-13T13:29:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1009541120 bytes 2024-04-13T13:29:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1087027200 bytes 2024-04-13T13:29:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1087671296 bytes 2024-04-13T13:29:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1146407936 bytes 2024-04-13T13:29:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1183794176 bytes 2024-04-13T13:29:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:29:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:29:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:29:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_27 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_27 128M 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2024-04-13T13:29:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:29:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-04-13T13:29:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:29:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-04-13T13:29:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:29:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_27': 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13765d22ecc8 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13765d22ecc8 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:17 2024 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:17 2024 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:17 2024 2024-04-13T13:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 7a6330ae-57ed-417b-a82d-4d987787e23b 2024-04-13T13:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:29:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:29:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_27 2024-04-13T13:29:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:29:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=6 2024-04-13T13:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:29:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 29461504 bytes 2024-04-13T13:29:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 53032960 bytes 2024-04-13T13:29:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 126666752 bytes 2024-04-13T13:29:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 212023296 bytes 2024-04-13T13:29:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 312504320 bytes 2024-04-13T13:29:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 354850816 bytes 2024-04-13T13:29:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 414356480 bytes 2024-04-13T13:29:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 460368896 bytes 2024-04-13T13:29:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 552630272 bytes 2024-04-13T13:29:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 646430720 bytes 2024-04-13T13:29:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 675976192 bytes 2024-04-13T13:29:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 696222720 bytes 2024-04-13T13:29:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 752146432 bytes 2024-04-13T13:29:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 789710848 bytes 2024-04-13T13:29:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 811515904 bytes 2024-04-13T13:29:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 896974848 bytes 2024-04-13T13:29:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 897594368 bytes 2024-04-13T13:29:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 923133952 bytes 2024-04-13T13:29:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 993523712 bytes 2024-04-13T13:29:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 1017358336 bytes 2024-04-13T13:29:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 1124061184 bytes 2024-04-13T13:29:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1151494144 bytes 2024-04-13T13:29:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1170719744 bytes 2024-04-13T13:29:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1225276416 bytes 2024-04-13T13:29:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1251037184 bytes 2024-04-13T13:29:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1281267712 bytes 2024-04-13T13:29:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1323758592 bytes 2024-04-13T13:29:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1368629248 bytes 2024-04-13T13:29:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1408446464 bytes 2024-04-13T13:29:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1430046720 bytes 2024-04-13T13:29:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:29:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:29:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_28 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:29:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_28 128M 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2024-04-13T13:29:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:29:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-04-13T13:29:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:29:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-04-13T13:29:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:29:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-04-13T13:29:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_28': 2024-04-13T13:29:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:29:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:29:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 137fd75dc63d 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.137fd75dc63d 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:24 2024 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:24 2024 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:24 2024 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 6c4ef21c-2a7c-465b-b23d-96050d4d8338 2024-04-13T13:29:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:29:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:29:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_28 2024-04-13T13:29:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:29:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:29:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:29:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:29:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:29:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 51298304 bytes 2024-04-13T13:29:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 91792384 bytes 2024-04-13T13:29:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 147311616 bytes 2024-04-13T13:29:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 153687040 bytes 2024-04-13T13:29:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 210115584 bytes 2024-04-13T13:29:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 264568832 bytes 2024-04-13T13:29:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 265160704 bytes 2024-04-13T13:29:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 304159744 bytes 2024-04-13T13:29:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 336079872 bytes 2024-04-13T13:29:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 366482432 bytes 2024-04-13T13:29:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 453022720 bytes 2024-04-13T13:29:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 514883584 bytes 2024-04-13T13:29:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 580978688 bytes 2024-04-13T13:29:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 633592832 bytes 2024-04-13T13:29:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 663066624 bytes 2024-04-13T13:29:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 718464000 bytes 2024-04-13T13:29:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 772126720 bytes 2024-04-13T13:29:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 814663680 bytes 2024-04-13T13:29:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 843171840 bytes 2024-04-13T13:29:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 872544256 bytes 2024-04-13T13:29:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 992319488 bytes 2024-04-13T13:29:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1019361280 bytes 2024-04-13T13:29:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1107191808 bytes 2024-04-13T13:29:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1176550400 bytes 2024-04-13T13:29:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1200370688 bytes 2024-04-13T13:29:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1253284864 bytes 2024-04-13T13:29:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1275219968 bytes 2024-04-13T13:29:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1323020288 bytes 2024-04-13T13:29:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1376476160 bytes 2024-04-13T13:29:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1441085440 bytes 2024-04-13T13:29:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1465018368 bytes 2024-04-13T13:29:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1465638912 bytes 2024-04-13T13:29:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1507947520 bytes 2024-04-13T13:29:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1555173376 bytes 2024-04-13T13:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1597418496 bytes 2024-04-13T13:29:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1665995776 bytes 2024-04-13T13:29:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1712090112 bytes 2024-04-13T13:29:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1731805184 bytes 2024-04-13T13:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1761748992 bytes 2024-04-13T13:29:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1797716992 bytes 2024-04-13T13:29:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1800355840 bytes 2024-04-13T13:29:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1860258816 bytes 2024-04-13T13:29:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1865796608 bytes 2024-04-13T13:29:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1933555712 bytes 2024-04-13T13:29:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1993373696 bytes 2024-04-13T13:29:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2001624064 bytes 2024-04-13T13:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2052661248 bytes 2024-04-13T13:29:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2111551488 bytes 2024-04-13T13:29:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2125278208 bytes 2024-04-13T13:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2177093632 bytes 2024-04-13T13:29:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2201682944 bytes 2024-04-13T13:29:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2265010176 bytes 2024-04-13T13:29:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2337143808 bytes 2024-04-13T13:29:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2350172160 bytes 2024-04-13T13:29:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2391113728 bytes 2024-04-13T13:29:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2454725632 bytes 2024-04-13T13:29:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2511842304 bytes 2024-04-13T13:29:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2537473024 bytes 2024-04-13T13:29:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2593384448 bytes 2024-04-13T13:29:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2607015936 bytes 2024-04-13T13:29:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2642893824 bytes 2024-04-13T13:29:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2658223104 bytes 2024-04-13T13:29:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2703634432 bytes 2024-04-13T13:29:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2761230336 bytes 2024-04-13T13:29:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2809656320 bytes 2024-04-13T13:29:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2865937408 bytes 2024-04-13T13:29:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2941323264 bytes 2024-04-13T13:29:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2966406144 bytes 2024-04-13T13:29:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_29 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_29 128M 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2024-04-13T13:29:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:29:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-04-13T13:29:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:29:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-04-13T13:29:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:29:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-04-13T13:29:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_29': 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 138ac24b01aa 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.138ac24b01aa 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:37 2024 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:37 2024 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:37 2024 2024-04-13T13:29:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:29:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:29:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: d1a11424-da29-4237-abbf-b187574ee6bd 2024-04-13T13:29:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:29:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:29:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_29 2024-04-13T13:29:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:29:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:29:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=11 2024-04-13T13:29:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:29:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:29:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 13491200 bytes 2024-04-13T13:29:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 48764928 bytes 2024-04-13T13:29:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 91724800 bytes 2024-04-13T13:29:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 116144128 bytes 2024-04-13T13:29:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 166051840 bytes 2024-04-13T13:29:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 258333696 bytes 2024-04-13T13:29:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 275055616 bytes 2024-04-13T13:29:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 322148352 bytes 2024-04-13T13:29:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 323375104 bytes 2024-04-13T13:29:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 403072000 bytes 2024-04-13T13:29:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 447047680 bytes 2024-04-13T13:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 554984448 bytes 2024-04-13T13:29:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 589856768 bytes 2024-04-13T13:29:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 651393024 bytes 2024-04-13T13:29:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 675815424 bytes 2024-04-13T13:29:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 676455424 bytes 2024-04-13T13:29:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 759009280 bytes 2024-04-13T13:29:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 760480768 bytes 2024-04-13T13:29:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 801511424 bytes 2024-04-13T13:29:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 820216832 bytes 2024-04-13T13:29:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 865520640 bytes 2024-04-13T13:29:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 938488832 bytes 2024-04-13T13:29:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 978120704 bytes 2024-04-13T13:29:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1014363136 bytes 2024-04-13T13:29:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1043809280 bytes 2024-04-13T13:29:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1071902720 bytes 2024-04-13T13:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1125800960 bytes 2024-04-13T13:29:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1174324224 bytes 2024-04-13T13:29:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1244549120 bytes 2024-04-13T13:29:42.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1252430848 bytes 2024-04-13T13:29:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1306928128 bytes 2024-04-13T13:29:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1392543744 bytes 2024-04-13T13:29:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1413470208 bytes 2024-04-13T13:29:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1436731392 bytes 2024-04-13T13:29:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1468247040 bytes 2024-04-13T13:29:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1514499072 bytes 2024-04-13T13:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1566175232 bytes 2024-04-13T13:29:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1576542208 bytes 2024-04-13T13:29:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1625692160 bytes 2024-04-13T13:29:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1700444160 bytes 2024-04-13T13:29:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1760822272 bytes 2024-04-13T13:29:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1848047616 bytes 2024-04-13T13:29:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1911314432 bytes 2024-04-13T13:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1987325952 bytes 2024-04-13T13:29:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 2056741888 bytes 2024-04-13T13:29:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2079811584 bytes 2024-04-13T13:29:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2112280576 bytes 2024-04-13T13:29:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2156890112 bytes 2024-04-13T13:29:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2216155136 bytes 2024-04-13T13:29:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2263651328 bytes 2024-04-13T13:29:46.816 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9756625845192917) > weight (0.25) 2024-04-13T13:29:46.816 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-13T13:29:46.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-13T13:29:46.817 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-13T13:29:46.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:2024-04-13T13:29:46.824+0000 7f751adf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 48640) UID: 1000 2024-04-13T13:29:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2364909568 bytes 2024-04-13T13:29:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2457141248 bytes 2024-04-13T13:29:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2522894336 bytes 2024-04-13T13:29:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2597891072 bytes 2024-04-13T13:29:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2623532032 bytes 2024-04-13T13:29:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2682575872 bytes 2024-04-13T13:29:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2697984000 bytes 2024-04-13T13:29:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2716753920 bytes 2024-04-13T13:29:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2821739520 bytes 2024-04-13T13:29:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:29:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:29:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:29:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:29:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_30 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_30 128M 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2024-04-13T13:29:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:29:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-04-13T13:29:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:29:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-04-13T13:29:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:29:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-04-13T13:29:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_30': 2024-04-13T13:29:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:29:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:29:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:29:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1394427205ca 2024-04-13T13:29:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1394427205ca 2024-04-13T13:29:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:29:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:48 2024 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:48 2024 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:48 2024 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: df429b8a-ec65-47e3-b367-41f75bbe4da1 2024-04-13T13:29:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_30 2024-04-13T13:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:29:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=8 2024-04-13T13:29:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:29:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:29:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 29497344 bytes 2024-04-13T13:29:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 72124416 bytes 2024-04-13T13:29:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 135905280 bytes 2024-04-13T13:29:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 175665152 bytes 2024-04-13T13:29:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 216655872 bytes 2024-04-13T13:29:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 217256960 bytes 2024-04-13T13:29:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 247586816 bytes 2024-04-13T13:29:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 307540992 bytes 2024-04-13T13:29:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 340355072 bytes 2024-04-13T13:29:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 363382784 bytes 2024-04-13T13:29:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 399992832 bytes 2024-04-13T13:29:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 411941888 bytes 2024-04-13T13:29:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 495491072 bytes 2024-04-13T13:29:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 551051264 bytes 2024-04-13T13:29:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 617482240 bytes 2024-04-13T13:29:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 639082496 bytes 2024-04-13T13:29:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 661362688 bytes 2024-04-13T13:29:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 668141568 bytes 2024-04-13T13:29:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 671332352 bytes 2024-04-13T13:29:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 695291904 bytes 2024-04-13T13:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 793375744 bytes 2024-04-13T13:29:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 853644288 bytes 2024-04-13T13:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 858588160 bytes 2024-04-13T13:29:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 892821504 bytes 2024-04-13T13:29:53.818 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-13T13:29:53.818 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:29:53.818 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-13T13:29:53.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-13T13:29:53.819 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-13T13:29:53.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-13T13:29:53.821 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-04-13T13:29:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 945895424 bytes 2024-04-13T13:29:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 986169344 bytes 2024-04-13T13:29:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1030343680 bytes 2024-04-13T13:29:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1036368896 bytes 2024-04-13T13:29:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1057270784 bytes 2024-04-13T13:29:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1078924288 bytes 2024-04-13T13:29:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1164052480 bytes 2024-04-13T13:29:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1208504320 bytes 2024-04-13T13:29:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1249679360 bytes 2024-04-13T13:29:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1253881856 bytes 2024-04-13T13:29:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1281280000 bytes 2024-04-13T13:29:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1335877632 bytes 2024-04-13T13:29:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1358882816 bytes 2024-04-13T13:29:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1372998656 bytes 2024-04-13T13:29:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1398907904 bytes 2024-04-13T13:29:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1432873984 bytes 2024-04-13T13:29:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1505850368 bytes 2024-04-13T13:29:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1520359424 bytes 2024-04-13T13:29:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1522801664 bytes 2024-04-13T13:29:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1551680512 bytes 2024-04-13T13:29:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1625389056 bytes 2024-04-13T13:29:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1649913856 bytes 2024-04-13T13:29:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1685641216 bytes 2024-04-13T13:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1749191680 bytes 2024-04-13T13:29:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_31 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_31 128M 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2024-04-13T13:29:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:29:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-04-13T13:29:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:29:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:29:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-04-13T13:29:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:29:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-04-13T13:29:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_31': 2024-04-13T13:29:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:29:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:29:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:29:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13a41daa9aa6 2024-04-13T13:29:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13a41daa9aa6 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:57 2024 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:57 2024 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:57 2024 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: f666e830-e8d8-4600-a27f-f8027ab46f85 2024-04-13T13:29:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:29:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:29:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_31 2024-04-13T13:29:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:29:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:29:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:29:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:29:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=11 2024-04-13T13:29:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:29:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:29:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 42728448 bytes 2024-04-13T13:29:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 116416512 bytes 2024-04-13T13:29:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 117038080 bytes 2024-04-13T13:29:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 193999872 bytes 2024-04-13T13:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 246303744 bytes 2024-04-13T13:29:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 274199552 bytes 2024-04-13T13:30:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 279334912 bytes 2024-04-13T13:30:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 297292800 bytes 2024-04-13T13:30:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 336886784 bytes 2024-04-13T13:30:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 340598784 bytes 2024-04-13T13:30:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 406161408 bytes 2024-04-13T13:30:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 422309888 bytes 2024-04-13T13:30:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 446172160 bytes 2024-04-13T13:30:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 465876992 bytes 2024-04-13T13:30:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 499960832 bytes 2024-04-13T13:30:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 557749248 bytes 2024-04-13T13:30:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 633032704 bytes 2024-04-13T13:30:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 652275712 bytes 2024-04-13T13:30:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 688115712 bytes 2024-04-13T13:30:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 720698368 bytes 2024-04-13T13:30:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 743542784 bytes 2024-04-13T13:30:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 783525888 bytes 2024-04-13T13:30:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 802988032 bytes 2024-04-13T13:30:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 833469440 bytes 2024-04-13T13:30:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 906430464 bytes 2024-04-13T13:30:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 931516416 bytes 2024-04-13T13:30:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 965700608 bytes 2024-04-13T13:30:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1023117312 bytes 2024-04-13T13:30:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1037675520 bytes 2024-04-13T13:30:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1080961024 bytes 2024-04-13T13:30:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1126798336 bytes 2024-04-13T13:30:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1160516608 bytes 2024-04-13T13:30:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1224544256 bytes 2024-04-13T13:30:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1284687872 bytes 2024-04-13T13:30:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1292919808 bytes 2024-04-13T13:30:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1360995328 bytes 2024-04-13T13:30:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1408097280 bytes 2024-04-13T13:30:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1472279552 bytes 2024-04-13T13:30:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1516492800 bytes 2024-04-13T13:30:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1535555584 bytes 2024-04-13T13:30:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1574615040 bytes 2024-04-13T13:30:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1575223296 bytes 2024-04-13T13:30:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1604950016 bytes 2024-04-13T13:30:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1670471680 bytes 2024-04-13T13:30:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1689814016 bytes 2024-04-13T13:30:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1690907648 bytes 2024-04-13T13:30:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1783078912 bytes 2024-04-13T13:30:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1837596672 bytes 2024-04-13T13:30:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1863108608 bytes 2024-04-13T13:30:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1878073344 bytes 2024-04-13T13:30:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1930938368 bytes 2024-04-13T13:30:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1988752384 bytes 2024-04-13T13:30:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2031462400 bytes 2024-04-13T13:30:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2064251904 bytes 2024-04-13T13:30:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2104003584 bytes 2024-04-13T13:30:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2149403648 bytes 2024-04-13T13:30:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2188652544 bytes 2024-04-13T13:30:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2213329920 bytes 2024-04-13T13:30:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2232839168 bytes 2024-04-13T13:30:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2283788288 bytes 2024-04-13T13:30:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2363952128 bytes 2024-04-13T13:30:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2427113472 bytes 2024-04-13T13:30:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2465802240 bytes 2024-04-13T13:30:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2470097920 bytes 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_32 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_32 128M 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:30:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:30:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2024-04-13T13:30:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:30:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-04-13T13:30:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:30:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-04-13T13:30:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:30:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-04-13T13:30:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_32': 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13ac136bf237 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13ac136bf237 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:09 2024 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:09 2024 2024-04-13T13:30:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:09 2024 2024-04-13T13:30:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:30:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:30:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: c95975b5-0eee-4f76-8be8-10a90b792c79 2024-04-13T13:30:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:30:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:30:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_32 2024-04-13T13:30:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:30:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:30:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=10 2024-04-13T13:30:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:30:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:30:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 51717120 bytes 2024-04-13T13:30:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 119478272 bytes 2024-04-13T13:30:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 141729792 bytes 2024-04-13T13:30:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 177780736 bytes 2024-04-13T13:30:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 258561024 bytes 2024-04-13T13:30:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 296287232 bytes 2024-04-13T13:30:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 324668416 bytes 2024-04-13T13:30:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 334978048 bytes 2024-04-13T13:30:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 353833984 bytes 2024-04-13T13:30:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 370438144 bytes 2024-04-13T13:30:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 400765952 bytes 2024-04-13T13:30:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 408129536 bytes 2024-04-13T13:30:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 467257344 bytes 2024-04-13T13:30:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 488370176 bytes 2024-04-13T13:30:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 554085376 bytes 2024-04-13T13:30:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 583388160 bytes 2024-04-13T13:30:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 617663488 bytes 2024-04-13T13:30:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 655868928 bytes 2024-04-13T13:30:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 656472064 bytes 2024-04-13T13:30:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 680064000 bytes 2024-04-13T13:30:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 719008768 bytes 2024-04-13T13:30:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 734560256 bytes 2024-04-13T13:30:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 789374976 bytes 2024-04-13T13:30:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 874028032 bytes 2024-04-13T13:30:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 921986048 bytes 2024-04-13T13:30:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 928932864 bytes 2024-04-13T13:30:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1010136064 bytes 2024-04-13T13:30:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1054532608 bytes 2024-04-13T13:30:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1067440128 bytes 2024-04-13T13:30:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1091346432 bytes 2024-04-13T13:30:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1092318208 bytes 2024-04-13T13:30:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1132061696 bytes 2024-04-13T13:30:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1204199424 bytes 2024-04-13T13:30:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1266536448 bytes 2024-04-13T13:30:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1327011840 bytes 2024-04-13T13:30:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1401457664 bytes 2024-04-13T13:30:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1455142912 bytes 2024-04-13T13:30:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1474751488 bytes 2024-04-13T13:30:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1542440960 bytes 2024-04-13T13:30:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1611751424 bytes 2024-04-13T13:30:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1668225024 bytes 2024-04-13T13:30:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1695700992 bytes 2024-04-13T13:30:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1742370816 bytes 2024-04-13T13:30:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1762494464 bytes 2024-04-13T13:30:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1852273664 bytes 2024-04-13T13:30:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1898600448 bytes 2024-04-13T13:30:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1924134912 bytes 2024-04-13T13:30:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1960306688 bytes 2024-04-13T13:30:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2004753408 bytes 2024-04-13T13:30:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2025042944 bytes 2024-04-13T13:30:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2096985088 bytes 2024-04-13T13:30:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2173377536 bytes 2024-04-13T13:30:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2243436544 bytes 2024-04-13T13:30:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2259034112 bytes 2024-04-13T13:30:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2271090688 bytes 2024-04-13T13:30:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2317720576 bytes 2024-04-13T13:30:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:30:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:30:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:30:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:30:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:30:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_33 2024-04-13T13:30:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_33 128M 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:30:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:30:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:30:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2024-04-13T13:30:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:30:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-04-13T13:30:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:30:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-04-13T13:30:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:30:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-04-13T13:30:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_33': 2024-04-13T13:30:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:30:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:30:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:30:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13b41f905dbc 2024-04-13T13:30:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13b41f905dbc 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:20 2024 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:20 2024 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:20 2024 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 51d2605d-d07c-452b-98c9-76a5433b1837 2024-04-13T13:30:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:30:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:30:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_33 2024-04-13T13:30:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:30:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:30:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=8 2024-04-13T13:30:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:30:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:30:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 57638912 bytes 2024-04-13T13:30:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 143994880 bytes 2024-04-13T13:30:22.717 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-13T13:30:22.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-13T13:30:22.718 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-13T13:30:22.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr:2024-04-13T13:30:22.725+0000 7fbebf1f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48279) UID: 1000 2024-04-13T13:30:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 262484992 bytes 2024-04-13T13:30:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 317273088 bytes 2024-04-13T13:30:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 360666112 bytes 2024-04-13T13:30:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 416846848 bytes 2024-04-13T13:30:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 466818048 bytes 2024-04-13T13:30:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 478395392 bytes 2024-04-13T13:30:23.718 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-13T13:30:23.718 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:30:23.719 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-13T13:30:23.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-13T13:30:23.719 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-13T13:30:23.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-13T13:30:23.722 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-04-13T13:30:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 501439488 bytes 2024-04-13T13:30:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 551806976 bytes 2024-04-13T13:30:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 552449024 bytes 2024-04-13T13:30:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 603993088 bytes 2024-04-13T13:30:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 640695296 bytes 2024-04-13T13:30:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 732355584 bytes 2024-04-13T13:30:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 774407168 bytes 2024-04-13T13:30:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 825886720 bytes 2024-04-13T13:30:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 895620096 bytes 2024-04-13T13:30:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 910474240 bytes 2024-04-13T13:30:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 924313600 bytes 2024-04-13T13:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 937826304 bytes 2024-04-13T13:30:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 973680640 bytes 2024-04-13T13:30:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 999559168 bytes 2024-04-13T13:30:26.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1043556352 bytes 2024-04-13T13:30:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1077945344 bytes 2024-04-13T13:30:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1135350784 bytes 2024-04-13T13:30:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1201665024 bytes 2024-04-13T13:30:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1255424000 bytes 2024-04-13T13:30:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1302545408 bytes 2024-04-13T13:30:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1325423616 bytes 2024-04-13T13:30:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1350734848 bytes 2024-04-13T13:30:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1377570816 bytes 2024-04-13T13:30:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1406134272 bytes 2024-04-13T13:30:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1473979392 bytes 2024-04-13T13:30:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1528806400 bytes 2024-04-13T13:30:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1573012480 bytes 2024-04-13T13:30:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1613669376 bytes 2024-04-13T13:30:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1645146112 bytes 2024-04-13T13:30:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1650183168 bytes 2024-04-13T13:30:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1698753536 bytes 2024-04-13T13:30:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1761391616 bytes 2024-04-13T13:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1783382016 bytes 2024-04-13T13:30:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1827678208 bytes 2024-04-13T13:30:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1828308992 bytes 2024-04-13T13:30:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1880827904 bytes 2024-04-13T13:30:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:30:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_34 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_34 2024-04-13T13:30:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_34 128M 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_34 2024-04-13T13:30:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:30:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:30:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:30:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2024-04-13T13:30:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:30:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-04-13T13:30:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_34 2024-04-13T13:30:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-04-13T13:30:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:30:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-04-13T13:30:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_34': 2024-04-13T13:30:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:30:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13c6cbcf1096 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13c6cbcf1096 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:30 2024 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:30 2024 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:30 2024 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1f7e0537-71e2-4dcf-a741-47e1160880d2 2024-04-13T13:30:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:30:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:30:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_34 2024-04-13T13:30:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_34 2024-04-13T13:30:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:30:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=5 2024-04-13T13:30:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:30:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:30:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 30878720 bytes 2024-04-13T13:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 50673664 bytes 2024-04-13T13:30:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 58649600 bytes 2024-04-13T13:30:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 85965824 bytes 2024-04-13T13:30:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 143843328 bytes 2024-04-13T13:30:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 228899840 bytes 2024-04-13T13:30:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 245381120 bytes 2024-04-13T13:30:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 272647168 bytes 2024-04-13T13:30:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 354153472 bytes 2024-04-13T13:30:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 396204032 bytes 2024-04-13T13:30:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 425085952 bytes 2024-04-13T13:30:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 449074176 bytes 2024-04-13T13:30:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 470552576 bytes 2024-04-13T13:30:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 471161856 bytes 2024-04-13T13:30:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 485991424 bytes 2024-04-13T13:30:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 537325568 bytes 2024-04-13T13:30:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 588817408 bytes 2024-04-13T13:30:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 638352384 bytes 2024-04-13T13:30:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 662885376 bytes 2024-04-13T13:30:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 746836992 bytes 2024-04-13T13:30:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 806105088 bytes 2024-04-13T13:30:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 847806464 bytes 2024-04-13T13:30:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 908082176 bytes 2024-04-13T13:30:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 932490240 bytes 2024-04-13T13:30:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 984743936 bytes 2024-04-13T13:30:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1001119744 bytes 2024-04-13T13:30:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1041043456 bytes 2024-04-13T13:30:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1065604096 bytes 2024-04-13T13:30:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1082821632 bytes 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_35 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_35 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_35 128M 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_35 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:30:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:30:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:30:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2024-04-13T13:30:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:30:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-04-13T13:30:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_35 2024-04-13T13:30:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-04-13T13:30:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:30:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-04-13T13:30:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_35': 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13ce8a81578a 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13ce8a81578a 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:36 2024 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:36 2024 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:36 2024 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1dd9564b-377b-4a17-82b5-1bee324ef4f2 2024-04-13T13:30:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:30:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:30:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_35 2024-04-13T13:30:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_35 2024-04-13T13:30:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:30:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=7 2024-04-13T13:30:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:30:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:30:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 19223552 bytes 2024-04-13T13:30:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 74880000 bytes 2024-04-13T13:30:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 119799808 bytes 2024-04-13T13:30:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 165030912 bytes 2024-04-13T13:30:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 186149888 bytes 2024-04-13T13:30:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 224521216 bytes 2024-04-13T13:30:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 279329792 bytes 2024-04-13T13:30:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 364398592 bytes 2024-04-13T13:30:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 399026176 bytes 2024-04-13T13:30:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 476333056 bytes 2024-04-13T13:30:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 530114560 bytes 2024-04-13T13:30:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 572959744 bytes 2024-04-13T13:30:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 715429888 bytes 2024-04-13T13:30:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 732441600 bytes 2024-04-13T13:30:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 752434176 bytes 2024-04-13T13:30:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 799628288 bytes 2024-04-13T13:30:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 869819392 bytes 2024-04-13T13:30:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 957389824 bytes 2024-04-13T13:30:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 978904064 bytes 2024-04-13T13:30:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 1013941248 bytes 2024-04-13T13:30:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 1040416768 bytes 2024-04-13T13:30:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1104457728 bytes 2024-04-13T13:30:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1158068224 bytes 2024-04-13T13:30:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1186621440 bytes 2024-04-13T13:30:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1228456960 bytes 2024-04-13T13:30:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1300912128 bytes 2024-04-13T13:30:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1351732224 bytes 2024-04-13T13:30:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1396045824 bytes 2024-04-13T13:30:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1426502656 bytes 2024-04-13T13:30:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1501228032 bytes 2024-04-13T13:30:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1527638016 bytes 2024-04-13T13:30:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1577158656 bytes 2024-04-13T13:30:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1664160768 bytes 2024-04-13T13:30:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1684164608 bytes 2024-04-13T13:30:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1736752128 bytes 2024-04-13T13:30:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1760040960 bytes 2024-04-13T13:30:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_36 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_36 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_36 128M 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_36 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2024-04-13T13:30:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:30:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-04-13T13:30:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_36 2024-04-13T13:30:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-04-13T13:30:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:30:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-04-13T13:30:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_36': 2024-04-13T13:30:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:30:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:30:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:30:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13d633fa43a2 2024-04-13T13:30:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13d633fa43a2 2024-04-13T13:30:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:30:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:30:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:30:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:30:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:44 2024 2024-04-13T13:30:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:44 2024 2024-04-13T13:30:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:44 2024 2024-04-13T13:30:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:30:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:30:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 37c03058-f2ca-48b7-b47d-ccfcb255694b 2024-04-13T13:30:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:30:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:30:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_36 2024-04-13T13:30:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_36 2024-04-13T13:30:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:30:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=11 2024-04-13T13:30:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:30:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:30:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 13681664 bytes 2024-04-13T13:30:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 40239104 bytes 2024-04-13T13:30:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 89192448 bytes 2024-04-13T13:30:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 109428736 bytes 2024-04-13T13:30:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 186312704 bytes 2024-04-13T13:30:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 231661568 bytes 2024-04-13T13:30:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 293258240 bytes 2024-04-13T13:30:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 301765632 bytes 2024-04-13T13:30:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 323172352 bytes 2024-04-13T13:30:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 383709184 bytes 2024-04-13T13:30:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 465112064 bytes 2024-04-13T13:30:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 541161472 bytes 2024-04-13T13:30:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 575842304 bytes 2024-04-13T13:30:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 584948736 bytes 2024-04-13T13:30:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 655306752 bytes 2024-04-13T13:30:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 749794304 bytes 2024-04-13T13:30:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 809547776 bytes 2024-04-13T13:30:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 915391488 bytes 2024-04-13T13:30:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 957762560 bytes 2024-04-13T13:30:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 1019640832 bytes 2024-04-13T13:30:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 1056565248 bytes 2024-04-13T13:30:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1082001408 bytes 2024-04-13T13:30:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1082633216 bytes 2024-04-13T13:30:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1125522432 bytes 2024-04-13T13:30:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1194497024 bytes 2024-04-13T13:30:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1225559040 bytes 2024-04-13T13:30:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1274180608 bytes 2024-04-13T13:30:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1312322560 bytes 2024-04-13T13:30:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1312935936 bytes 2024-04-13T13:30:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1340349440 bytes 2024-04-13T13:30:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1350568960 bytes 2024-04-13T13:30:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1408463872 bytes 2024-04-13T13:30:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1450629120 bytes 2024-04-13T13:30:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1525310464 bytes 2024-04-13T13:30:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1584223232 bytes 2024-04-13T13:30:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1635616768 bytes 2024-04-13T13:30:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1704148992 bytes 2024-04-13T13:30:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1761979392 bytes 2024-04-13T13:30:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1844371456 bytes 2024-04-13T13:30:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1907501056 bytes 2024-04-13T13:30:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1919186944 bytes 2024-04-13T13:30:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1952867328 bytes 2024-04-13T13:30:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1963869184 bytes 2024-04-13T13:30:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 2028704768 bytes 2024-04-13T13:30:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 2048753664 bytes 2024-04-13T13:30:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2158675968 bytes 2024-04-13T13:30:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2196945920 bytes 2024-04-13T13:30:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2251360256 bytes 2024-04-13T13:30:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2259060736 bytes 2024-04-13T13:30:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2317880320 bytes 2024-04-13T13:30:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2369349632 bytes 2024-04-13T13:30:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2418580480 bytes 2024-04-13T13:30:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2463527936 bytes 2024-04-13T13:30:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2552700928 bytes 2024-04-13T13:30:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2595905536 bytes 2024-04-13T13:30:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2596523008 bytes 2024-04-13T13:30:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2697203712 bytes 2024-04-13T13:30:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2731745280 bytes 2024-04-13T13:30:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2746607616 bytes 2024-04-13T13:30:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:30:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_37 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_37 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_37 128M 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_37 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:30:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2024-04-13T13:30:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:30:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-04-13T13:30:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_37 2024-04-13T13:30:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:30:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-04-13T13:30:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:30:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-04-13T13:30:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_37': 2024-04-13T13:30:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:30:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:30:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:30:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13e0970a5ef6 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13e0970a5ef6 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:56 2024 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:56 2024 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:56 2024 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: b1d1e9b9-23ac-4857-a6a7-e354f28ab5cd 2024-04-13T13:30:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:30:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:30:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_37 2024-04-13T13:30:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:30:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:30:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_37 2024-04-13T13:30:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:30:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=14 2024-04-13T13:30:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:30:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:30:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 617472 bytes 2024-04-13T13:30:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 75132928 bytes 2024-04-13T13:30:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 132824064 bytes 2024-04-13T13:30:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 149215232 bytes 2024-04-13T13:30:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 172725248 bytes 2024-04-13T13:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 200620032 bytes 2024-04-13T13:30:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 202038272 bytes 2024-04-13T13:30:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 202644480 bytes 2024-04-13T13:30:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 216799232 bytes 2024-04-13T13:30:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 278858752 bytes 2024-04-13T13:30:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 311103488 bytes 2024-04-13T13:30:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 344989696 bytes 2024-04-13T13:30:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 402903040 bytes 2024-04-13T13:30:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 449483776 bytes 2024-04-13T13:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 477902848 bytes 2024-04-13T13:30:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 520415232 bytes 2024-04-13T13:30:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 552423424 bytes 2024-04-13T13:30:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 558785536 bytes 2024-04-13T13:31:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 596627456 bytes 2024-04-13T13:31:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 617867264 bytes 2024-04-13T13:31:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 623699968 bytes 2024-04-13T13:31:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 643502080 bytes 2024-04-13T13:31:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 690981888 bytes 2024-04-13T13:31:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 818527232 bytes 2024-04-13T13:31:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 885615616 bytes 2024-04-13T13:31:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 921194496 bytes 2024-04-13T13:31:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1010976768 bytes 2024-04-13T13:31:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1030537216 bytes 2024-04-13T13:31:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1056894976 bytes 2024-04-13T13:31:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1095204864 bytes 2024-04-13T13:31:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1136214016 bytes 2024-04-13T13:31:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1214953472 bytes 2024-04-13T13:31:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1247892480 bytes 2024-04-13T13:31:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1282348032 bytes 2024-04-13T13:31:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1317348352 bytes 2024-04-13T13:31:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1407296512 bytes 2024-04-13T13:31:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1506073600 bytes 2024-04-13T13:31:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1532828672 bytes 2024-04-13T13:31:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1601579008 bytes 2024-04-13T13:31:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1649239040 bytes 2024-04-13T13:31:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1693464576 bytes 2024-04-13T13:31:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1762285568 bytes 2024-04-13T13:31:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1806307328 bytes 2024-04-13T13:31:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1828584448 bytes 2024-04-13T13:31:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1883486208 bytes 2024-04-13T13:31:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1906021376 bytes 2024-04-13T13:31:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1961595904 bytes 2024-04-13T13:31:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1990222848 bytes 2024-04-13T13:31:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2008525824 bytes 2024-04-13T13:31:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2092917760 bytes 2024-04-13T13:31:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2104303616 bytes 2024-04-13T13:31:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2117349376 bytes 2024-04-13T13:31:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2212519936 bytes 2024-04-13T13:31:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2311710720 bytes 2024-04-13T13:31:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2434337792 bytes 2024-04-13T13:31:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2449510400 bytes 2024-04-13T13:31:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2510709760 bytes 2024-04-13T13:31:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2566019072 bytes 2024-04-13T13:31:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2570003456 bytes 2024-04-13T13:31:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2594629632 bytes 2024-04-13T13:31:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2613917696 bytes 2024-04-13T13:31:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2652488704 bytes 2024-04-13T13:31:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2697283584 bytes 2024-04-13T13:31:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2732182528 bytes 2024-04-13T13:31:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2782337024 bytes 2024-04-13T13:31:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2830622720 bytes 2024-04-13T13:31:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2909315072 bytes 2024-04-13T13:31:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2971830272 bytes 2024-04-13T13:31:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 3008826368 bytes 2024-04-13T13:31:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 3114867712 bytes 2024-04-13T13:31:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 3116148736 bytes 2024-04-13T13:31:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 3139236864 bytes 2024-04-13T13:31:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 3209320448 bytes 2024-04-13T13:31:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 3237740544 bytes 2024-04-13T13:31:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 3321964544 bytes 2024-04-13T13:31:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3375090688 bytes 2024-04-13T13:31:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:31:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:31:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:31:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_38 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_38 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_38 128M 2024-04-13T13:31:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_38 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2024-04-13T13:31:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:31:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-04-13T13:31:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_38 2024-04-13T13:31:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:31:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-04-13T13:31:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:31:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_38': 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13e9904619b0 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13e9904619b0 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:31:11 2024 2024-04-13T13:31:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:31:11 2024 2024-04-13T13:31:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:31:11 2024 2024-04-13T13:31:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:31:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:31:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: d96b2a28-af93-4496-b387-393648c236e0 2024-04-13T13:31:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:31:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:31:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_38 2024-04-13T13:31:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_38 2024-04-13T13:31:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:31:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=14 2024-04-13T13:31:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:31:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:31:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 105439232 bytes 2024-04-13T13:31:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 148349952 bytes 2024-04-13T13:31:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 156994560 bytes 2024-04-13T13:31:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 170067968 bytes 2024-04-13T13:31:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 221355008 bytes 2024-04-13T13:31:12.705 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:31:12.712+0000 7f59186a3640 -1 reset not still connected to 0x55ea40f15860 2024-04-13T13:31:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 272968704 bytes 2024-04-13T13:31:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 283383808 bytes 2024-04-13T13:31:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 305892352 bytes 2024-04-13T13:31:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 378546176 bytes 2024-04-13T13:31:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 403968000 bytes 2024-04-13T13:31:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 416173056 bytes 2024-04-13T13:31:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 458270720 bytes 2024-04-13T13:31:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 494545920 bytes 2024-04-13T13:31:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 552485888 bytes 2024-04-13T13:31:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 582033408 bytes 2024-04-13T13:31:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 624467968 bytes 2024-04-13T13:31:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 675888128 bytes 2024-04-13T13:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 754675712 bytes 2024-04-13T13:31:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 775586816 bytes 2024-04-13T13:31:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 791546880 bytes 2024-04-13T13:31:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 838012928 bytes 2024-04-13T13:31:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 849173504 bytes 2024-04-13T13:31:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 904015872 bytes 2024-04-13T13:31:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 925049856 bytes 2024-04-13T13:31:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 979491840 bytes 2024-04-13T13:31:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1005034496 bytes 2024-04-13T13:31:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1056613376 bytes 2024-04-13T13:31:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1074327552 bytes 2024-04-13T13:31:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1110719488 bytes 2024-04-13T13:31:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1199767552 bytes 2024-04-13T13:31:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1213763584 bytes 2024-04-13T13:31:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1289676800 bytes 2024-04-13T13:31:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1311881216 bytes 2024-04-13T13:31:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1337170944 bytes 2024-04-13T13:31:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1339937792 bytes 2024-04-13T13:31:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1397641216 bytes 2024-04-13T13:31:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1442779136 bytes 2024-04-13T13:31:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1507390464 bytes 2024-04-13T13:31:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1528476672 bytes 2024-04-13T13:31:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1561137152 bytes 2024-04-13T13:31:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1620274176 bytes 2024-04-13T13:31:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1620900864 bytes 2024-04-13T13:31:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1680140288 bytes 2024-04-13T13:31:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1716695040 bytes 2024-04-13T13:31:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1784240128 bytes 2024-04-13T13:31:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1823699968 bytes 2024-04-13T13:31:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1857145856 bytes 2024-04-13T13:31:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1910749184 bytes 2024-04-13T13:31:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1954481152 bytes 2024-04-13T13:31:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2004857856 bytes 2024-04-13T13:31:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2037617664 bytes 2024-04-13T13:31:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2107987968 bytes 2024-04-13T13:31:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2189239296 bytes 2024-04-13T13:31:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2234379264 bytes 2024-04-13T13:31:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2275089408 bytes 2024-04-13T13:31:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2316490752 bytes 2024-04-13T13:31:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2317096960 bytes 2024-04-13T13:31:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2350266368 bytes 2024-04-13T13:31:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2365327360 bytes 2024-04-13T13:31:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2439376896 bytes 2024-04-13T13:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2480067584 bytes 2024-04-13T13:31:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2594125824 bytes 2024-04-13T13:31:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2604126208 bytes 2024-04-13T13:31:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2638070784 bytes 2024-04-13T13:31:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2650954752 bytes 2024-04-13T13:31:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2679105536 bytes 2024-04-13T13:31:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2712809472 bytes 2024-04-13T13:31:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2720600064 bytes 2024-04-13T13:31:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2790551552 bytes 2024-04-13T13:31:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2833600512 bytes 2024-04-13T13:31:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2934137856 bytes 2024-04-13T13:31:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2957972480 bytes 2024-04-13T13:31:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 3004976128 bytes 2024-04-13T13:31:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 3036994560 bytes 2024-04-13T13:31:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 3073401856 bytes 2024-04-13T13:31:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3115056128 bytes 2024-04-13T13:31:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 3161104384 bytes 2024-04-13T13:31:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7800 IOs, 3212158976 bytes 2024-04-13T13:31:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_39 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_39 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_39 128M 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_39 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2024-04-13T13:31:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-04-13T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_39 2024-04-13T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-04-13T13:31:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:31:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-04-13T13:31:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_39': 2024-04-13T13:31:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:31:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:31:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13f1b83ef1a6 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13f1b83ef1a6 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:31:25 2024 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:31:25 2024 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:31:25 2024 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:31:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 41ea30c5-3df3-4041-8289-68ecbae98d63 2024-04-13T13:31:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:31:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:31:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_39 2024-04-13T13:31:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_39 2024-04-13T13:31:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:31:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:31:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:31:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:31:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 53072896 bytes 2024-04-13T13:31:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 142356480 bytes 2024-04-13T13:31:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 142981120 bytes 2024-04-13T13:31:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 171970560 bytes 2024-04-13T13:31:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 241593344 bytes 2024-04-13T13:31:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 291600384 bytes 2024-04-13T13:31:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 311081984 bytes 2024-04-13T13:31:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 374227968 bytes 2024-04-13T13:31:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 401016832 bytes 2024-04-13T13:31:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 452762624 bytes 2024-04-13T13:31:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 496470016 bytes 2024-04-13T13:31:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 582352896 bytes 2024-04-13T13:31:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 620001280 bytes 2024-04-13T13:31:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 669617152 bytes 2024-04-13T13:31:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 740326400 bytes 2024-04-13T13:31:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 770173952 bytes 2024-04-13T13:31:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 771055616 bytes 2024-04-13T13:31:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 775870464 bytes 2024-04-13T13:31:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 789779456 bytes 2024-04-13T13:31:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 895186944 bytes 2024-04-13T13:31:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 933620736 bytes 2024-04-13T13:31:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 986278912 bytes 2024-04-13T13:31:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1038896128 bytes 2024-04-13T13:31:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1071437824 bytes 2024-04-13T13:31:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1163777024 bytes 2024-04-13T13:31:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1215457280 bytes 2024-04-13T13:31:31.822 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7867658990187231) > weight (0.25) 2024-04-13T13:31:31.822 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9354455817279171) > weight (0.25) 2024-04-13T13:31:31.822 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3912000777994319) > weight (0.25) 2024-04-13T13:31:31.822 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5351375385959278) > weight (0.25) 2024-04-13T13:31:31.822 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-04-13T13:31:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1260481536 bytes 2024-04-13T13:31:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1282956288 bytes 2024-04-13T13:31:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1334067200 bytes 2024-04-13T13:31:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1358523392 bytes 2024-04-13T13:31:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1426770944 bytes 2024-04-13T13:31:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1536648192 bytes 2024-04-13T13:31:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1567051776 bytes 2024-04-13T13:31:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1628631040 bytes 2024-04-13T13:31:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1655584768 bytes 2024-04-13T13:31:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1659115520 bytes 2024-04-13T13:31:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1770119168 bytes 2024-04-13T13:31:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1797329920 bytes 2024-04-13T13:31:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1883441152 bytes 2024-04-13T13:31:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1914449920 bytes 2024-04-13T13:31:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1939947520 bytes 2024-04-13T13:31:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1974406144 bytes 2024-04-13T13:31:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1993090048 bytes 2024-04-13T13:31:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 2047878144 bytes 2024-04-13T13:31:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 2084812800 bytes 2024-04-13T13:31:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2118228992 bytes 2024-04-13T13:31:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2187048960 bytes 2024-04-13T13:31:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2213459968 bytes 2024-04-13T13:31:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2236817408 bytes 2024-04-13T13:31:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2262059008 bytes 2024-04-13T13:31:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2267863040 bytes 2024-04-13T13:31:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2273466368 bytes 2024-04-13T13:31:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2289205248 bytes 2024-04-13T13:31:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2299329536 bytes 2024-04-13T13:31:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2302818304 bytes 2024-04-13T13:31:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2318718976 bytes 2024-04-13T13:31:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2359149568 bytes 2024-04-13T13:31:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2361460736 bytes 2024-04-13T13:31:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2386806784 bytes 2024-04-13T13:31:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2434228224 bytes 2024-04-13T13:31:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2483975168 bytes 2024-04-13T13:31:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2570692608 bytes 2024-04-13T13:31:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2631340032 bytes 2024-04-13T13:31:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2696144896 bytes 2024-04-13T13:31:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2807226368 bytes 2024-04-13T13:31:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2850322432 bytes 2024-04-13T13:31:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:31:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:31:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:31:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:31:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:31:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_40 2024-04-13T13:31:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-04-13T13:31:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_40 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_40 128M 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_40 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:31:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:31:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:31:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2024-04-13T13:31:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:31:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-04-13T13:31:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_40 2024-04-13T13:31:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:31:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-04-13T13:31:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:31:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_40': 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 13f9856d3c0b 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.13f9856d3c0b 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:31:38 2024 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:31:38 2024 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:31:38 2024 2024-04-13T13:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:31:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:31:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8ed18fcc-ddf5-45b3-bceb-3cbe6226dcaa 2024-04-13T13:31:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:31:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:31:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_40 2024-04-13T13:31:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_40 2024-04-13T13:31:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:31:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=13 2024-04-13T13:31:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:31:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:31:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 37788672 bytes 2024-04-13T13:31:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 62001152 bytes 2024-04-13T13:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 148200448 bytes 2024-04-13T13:31:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 222457856 bytes 2024-04-13T13:31:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 254882816 bytes 2024-04-13T13:31:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 286951424 bytes 2024-04-13T13:31:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 346328064 bytes 2024-04-13T13:31:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 422875136 bytes 2024-04-13T13:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 457407488 bytes 2024-04-13T13:31:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 530484224 bytes 2024-04-13T13:31:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 589090816 bytes 2024-04-13T13:31:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 633801728 bytes 2024-04-13T13:31:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 669348864 bytes 2024-04-13T13:31:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 737413120 bytes 2024-04-13T13:31:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 820536320 bytes 2024-04-13T13:31:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 843738112 bytes 2024-04-13T13:31:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 844421120 bytes 2024-04-13T13:31:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 917852160 bytes 2024-04-13T13:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 997710848 bytes 2024-04-13T13:31:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 1043542016 bytes 2024-04-13T13:31:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 1062070272 bytes 2024-04-13T13:31:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1088107520 bytes 2024-04-13T13:31:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1103946752 bytes 2024-04-13T13:31:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1162280960 bytes 2024-04-13T13:31:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1220567040 bytes 2024-04-13T13:31:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1285107712 bytes 2024-04-13T13:31:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1314144256 bytes 2024-04-13T13:31:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1327694848 bytes 2024-04-13T13:31:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1398792192 bytes 2024-04-13T13:31:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1467483136 bytes 2024-04-13T13:31:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1510509568 bytes 2024-04-13T13:31:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1543283712 bytes 2024-04-13T13:31:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1570498560 bytes 2024-04-13T13:31:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1603952640 bytes 2024-04-13T13:31:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1638517760 bytes 2024-04-13T13:31:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1726558208 bytes 2024-04-13T13:31:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1766022144 bytes 2024-04-13T13:31:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1875274752 bytes 2024-04-13T13:31:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1901777920 bytes 2024-04-13T13:31:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1902375936 bytes 2024-04-13T13:31:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1931834368 bytes 2024-04-13T13:31:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1944692736 bytes 2024-04-13T13:31:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1982949376 bytes 2024-04-13T13:31:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 2021178368 bytes 2024-04-13T13:31:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 2106166272 bytes 2024-04-13T13:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2219012096 bytes 2024-04-13T13:31:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2219639808 bytes 2024-04-13T13:31:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2329801728 bytes 2024-04-13T13:31:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2371357696 bytes 2024-04-13T13:31:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2424637440 bytes 2024-04-13T13:31:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2490152960 bytes 2024-04-13T13:31:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2595613696 bytes 2024-04-13T13:31:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2689548288 bytes 2024-04-13T13:31:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2732271616 bytes 2024-04-13T13:31:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2789059584 bytes 2024-04-13T13:31:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2846344192 bytes 2024-04-13T13:31:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2891750400 bytes 2024-04-13T13:31:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2936563712 bytes 2024-04-13T13:31:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2962511872 bytes 2024-04-13T13:31:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2999101440 bytes 2024-04-13T13:31:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 3066674176 bytes 2024-04-13T13:31:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 3108858880 bytes 2024-04-13T13:31:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 3133654016 bytes 2024-04-13T13:31:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 3200011264 bytes 2024-04-13T13:31:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 3292098560 bytes 2024-04-13T13:31:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 3302533120 bytes 2024-04-13T13:31:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:31:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:31:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:31:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:31:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_41 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_41 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_41 128M 2024-04-13T13:31:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_41 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:31:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2024-04-13T13:31:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:31:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-04-13T13:31:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_41 2024-04-13T13:31:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:31:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-04-13T13:31:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:31:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-04-13T13:31:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_41': 2024-04-13T13:31:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:31:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:31:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:31:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 140298a58363 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.140298a58363 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:31:53 2024 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:31:53 2024 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:31:53 2024 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 682df14d-9783-44dd-8781-b0f0d506d592 2024-04-13T13:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:31:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:31:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_41 2024-04-13T13:31:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:31:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:31:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_41 2024-04-13T13:31:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:31:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=10 2024-04-13T13:31:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:31:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:31:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 39715840 bytes 2024-04-13T13:31:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 101820416 bytes 2024-04-13T13:31:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 164573184 bytes 2024-04-13T13:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 218255360 bytes 2024-04-13T13:31:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 254748672 bytes 2024-04-13T13:31:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 263693312 bytes 2024-04-13T13:31:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 301884416 bytes 2024-04-13T13:31:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 320489472 bytes 2024-04-13T13:31:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 334918656 bytes 2024-04-13T13:31:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 361631744 bytes 2024-04-13T13:31:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 386176000 bytes 2024-04-13T13:31:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 437633024 bytes 2024-04-13T13:31:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 496772096 bytes 2024-04-13T13:31:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 555563008 bytes 2024-04-13T13:31:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 590371840 bytes 2024-04-13T13:31:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 647494656 bytes 2024-04-13T13:31:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 665277440 bytes 2024-04-13T13:31:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 718676992 bytes 2024-04-13T13:31:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 737070080 bytes 2024-04-13T13:31:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 798225408 bytes 2024-04-13T13:31:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 833340416 bytes 2024-04-13T13:31:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 858114048 bytes 2024-04-13T13:31:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 885164032 bytes 2024-04-13T13:31:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 907802624 bytes 2024-04-13T13:31:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 985217024 bytes 2024-04-13T13:31:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1026469888 bytes 2024-04-13T13:31:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1034426368 bytes 2024-04-13T13:31:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1082447872 bytes 2024-04-13T13:31:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1139245056 bytes 2024-04-13T13:31:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1139862528 bytes 2024-04-13T13:31:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1182990336 bytes 2024-04-13T13:31:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1235343360 bytes 2024-04-13T13:31:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1266472960 bytes 2024-04-13T13:31:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1346676736 bytes 2024-04-13T13:32:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1366073344 bytes 2024-04-13T13:32:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1420329984 bytes 2024-04-13T13:32:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1457900544 bytes 2024-04-13T13:32:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1534851072 bytes 2024-04-13T13:32:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1644377088 bytes 2024-04-13T13:32:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1663307776 bytes 2024-04-13T13:32:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1684512768 bytes 2024-04-13T13:32:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1699427328 bytes 2024-04-13T13:32:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1751276544 bytes 2024-04-13T13:32:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1784480768 bytes 2024-04-13T13:32:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1838658560 bytes 2024-04-13T13:32:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1874053120 bytes 2024-04-13T13:32:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1943656448 bytes 2024-04-13T13:32:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1975353344 bytes 2024-04-13T13:32:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2014411776 bytes 2024-04-13T13:32:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2028649472 bytes 2024-04-13T13:32:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2067530752 bytes 2024-04-13T13:32:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2121127936 bytes 2024-04-13T13:32:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2180615168 bytes 2024-04-13T13:32:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2214019072 bytes 2024-04-13T13:32:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2261751808 bytes 2024-04-13T13:32:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2325016576 bytes 2024-04-13T13:32:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_42 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_42 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_42 128M 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_42 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2024-04-13T13:32:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:32:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-04-13T13:32:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_42 2024-04-13T13:32:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-04-13T13:32:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:32:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_42': 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 140b187170bc 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.140b187170bc 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:32:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:04 2024 2024-04-13T13:32:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:04 2024 2024-04-13T13:32:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:04 2024 2024-04-13T13:32:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:32:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:32:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: a05e5ef4-930c-4ff6-b504-51ec542751fd 2024-04-13T13:32:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:32:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:32:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_42 2024-04-13T13:32:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_42 2024-04-13T13:32:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:32:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=13 2024-04-13T13:32:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:32:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:32:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 27618304 bytes 2024-04-13T13:32:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 56032256 bytes 2024-04-13T13:32:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 86091776 bytes 2024-04-13T13:32:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 160017408 bytes 2024-04-13T13:32:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 196054016 bytes 2024-04-13T13:32:06.010 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:32:06.017+0000 7f59186a3640 -1 reset not still connected to 0x55ea1e897520 2024-04-13T13:32:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 235047936 bytes 2024-04-13T13:32:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 235646976 bytes 2024-04-13T13:32:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 277131264 bytes 2024-04-13T13:32:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 291233792 bytes 2024-04-13T13:32:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 307031040 bytes 2024-04-13T13:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 307640320 bytes 2024-04-13T13:32:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 361124864 bytes 2024-04-13T13:32:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 388454400 bytes 2024-04-13T13:32:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 431789056 bytes 2024-04-13T13:32:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 455114752 bytes 2024-04-13T13:32:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 515351552 bytes 2024-04-13T13:32:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 579063808 bytes 2024-04-13T13:32:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 595529728 bytes 2024-04-13T13:32:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 643802112 bytes 2024-04-13T13:32:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 675671040 bytes 2024-04-13T13:32:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 676255744 bytes 2024-04-13T13:32:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 710338560 bytes 2024-04-13T13:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 729923584 bytes 2024-04-13T13:32:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 741924864 bytes 2024-04-13T13:32:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 796406784 bytes 2024-04-13T13:32:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 812795904 bytes 2024-04-13T13:32:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 865455104 bytes 2024-04-13T13:32:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 893838336 bytes 2024-04-13T13:32:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 943367168 bytes 2024-04-13T13:32:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 995600384 bytes 2024-04-13T13:32:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1027318784 bytes 2024-04-13T13:32:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1057411072 bytes 2024-04-13T13:32:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1113755648 bytes 2024-04-13T13:32:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1213986816 bytes 2024-04-13T13:32:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1227032576 bytes 2024-04-13T13:32:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1279491072 bytes 2024-04-13T13:32:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1319877632 bytes 2024-04-13T13:32:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1320470528 bytes 2024-04-13T13:32:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1348321280 bytes 2024-04-13T13:32:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1436908544 bytes 2024-04-13T13:32:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1494215680 bytes 2024-04-13T13:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1572049920 bytes 2024-04-13T13:32:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1611025408 bytes 2024-04-13T13:32:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1667813376 bytes 2024-04-13T13:32:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1749563392 bytes 2024-04-13T13:32:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1788080128 bytes 2024-04-13T13:32:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1808203776 bytes 2024-04-13T13:32:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1856331776 bytes 2024-04-13T13:32:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1884024832 bytes 2024-04-13T13:32:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1884677120 bytes 2024-04-13T13:32:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1926102016 bytes 2024-04-13T13:32:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1936413696 bytes 2024-04-13T13:32:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 1980887040 bytes 2024-04-13T13:32:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2037534720 bytes 2024-04-13T13:32:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2103334912 bytes 2024-04-13T13:32:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2192577536 bytes 2024-04-13T13:32:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2242215936 bytes 2024-04-13T13:32:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2281766912 bytes 2024-04-13T13:32:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2333008896 bytes 2024-04-13T13:32:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2338018304 bytes 2024-04-13T13:32:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2392279040 bytes 2024-04-13T13:32:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2482327552 bytes 2024-04-13T13:32:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2561949696 bytes 2024-04-13T13:32:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2597998592 bytes 2024-04-13T13:32:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2646653952 bytes 2024-04-13T13:32:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2664673280 bytes 2024-04-13T13:32:16.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6752534392934841) > weight (0.25) 2024-04-13T13:32:16.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9121201982695932) > weight (0.25) 2024-04-13T13:32:16.723 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-13T13:32:16.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-13T13:32:16.723 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-13T13:32:16.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi122.stderr:2024-04-13T13:32:16.733+0000 7ff31fbf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 50683) UID: 1000 2024-04-13T13:32:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2713971712 bytes 2024-04-13T13:32:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2764591104 bytes 2024-04-13T13:32:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2823392256 bytes 2024-04-13T13:32:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2833130496 bytes 2024-04-13T13:32:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2868631552 bytes 2024-04-13T13:32:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2869221376 bytes 2024-04-13T13:32:17.724 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-13T13:32:17.724 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:32:17.724 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-13T13:32:17.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-13T13:32:17.725 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-13T13:32:17.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-13T13:32:17.727 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-04-13T13:32:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2872922112 bytes 2024-04-13T13:32:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 2891540480 bytes 2024-04-13T13:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 2915388416 bytes 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_43 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_43 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_43 128M 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_43 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:32:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:32:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:32:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2024-04-13T13:32:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:32:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-04-13T13:32:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_43 2024-04-13T13:32:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-04-13T13:32:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:32:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-04-13T13:32:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_43': 2024-04-13T13:32:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:32:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:32:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:32:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 141f6045c29f 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.141f6045c29f 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:18 2024 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:18 2024 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:18 2024 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: c0e8ec4e-880c-4448-9702-ba9518d9f196 2024-04-13T13:32:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:32:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:32:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_43 2024-04-13T13:32:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_43 2024-04-13T13:32:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:32:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:32:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:32:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:32:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 22378496 bytes 2024-04-13T13:32:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 48761856 bytes 2024-04-13T13:32:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 59794432 bytes 2024-04-13T13:32:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 110054400 bytes 2024-04-13T13:32:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 131423232 bytes 2024-04-13T13:32:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 198791168 bytes 2024-04-13T13:32:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 267996160 bytes 2024-04-13T13:32:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 288933888 bytes 2024-04-13T13:32:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 321943552 bytes 2024-04-13T13:32:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 401241088 bytes 2024-04-13T13:32:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 440599552 bytes 2024-04-13T13:32:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 500941824 bytes 2024-04-13T13:32:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 548687872 bytes 2024-04-13T13:32:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 615487488 bytes 2024-04-13T13:32:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 661139456 bytes 2024-04-13T13:32:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 695443456 bytes 2024-04-13T13:32:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 722544640 bytes 2024-04-13T13:32:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 734748672 bytes 2024-04-13T13:32:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 765808640 bytes 2024-04-13T13:32:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 804855808 bytes 2024-04-13T13:32:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 842504192 bytes 2024-04-13T13:32:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 877927424 bytes 2024-04-13T13:32:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 919652352 bytes 2024-04-13T13:32:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 952010752 bytes 2024-04-13T13:32:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 999839744 bytes 2024-04-13T13:32:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1018071040 bytes 2024-04-13T13:32:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1028621312 bytes 2024-04-13T13:32:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1062813696 bytes 2024-04-13T13:32:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1073309696 bytes 2024-04-13T13:32:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1096588288 bytes 2024-04-13T13:32:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1119506432 bytes 2024-04-13T13:32:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1122868224 bytes 2024-04-13T13:32:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1167499264 bytes 2024-04-13T13:32:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1197003776 bytes 2024-04-13T13:32:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1246707712 bytes 2024-04-13T13:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1300758528 bytes 2024-04-13T13:32:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1301374976 bytes 2024-04-13T13:32:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1356527616 bytes 2024-04-13T13:32:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1375673344 bytes 2024-04-13T13:32:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1379391488 bytes 2024-04-13T13:32:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1439739904 bytes 2024-04-13T13:32:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1469888512 bytes 2024-04-13T13:32:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1552720896 bytes 2024-04-13T13:32:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1598244864 bytes 2024-04-13T13:32:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1631381504 bytes 2024-04-13T13:32:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1673001984 bytes 2024-04-13T13:32:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1715809280 bytes 2024-04-13T13:32:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1721691136 bytes 2024-04-13T13:32:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1760354304 bytes 2024-04-13T13:32:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1805906944 bytes 2024-04-13T13:32:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1806513152 bytes 2024-04-13T13:32:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1866519552 bytes 2024-04-13T13:32:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 1909866496 bytes 2024-04-13T13:32:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 1992596480 bytes 2024-04-13T13:32:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2038028288 bytes 2024-04-13T13:32:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2049463296 bytes 2024-04-13T13:32:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2100112384 bytes 2024-04-13T13:32:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2147825664 bytes 2024-04-13T13:32:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2189771776 bytes 2024-04-13T13:32:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2284935168 bytes 2024-04-13T13:32:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2303455232 bytes 2024-04-13T13:32:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2384948224 bytes 2024-04-13T13:32:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2411966464 bytes 2024-04-13T13:32:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2443075584 bytes 2024-04-13T13:32:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2487325696 bytes 2024-04-13T13:32:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2501688320 bytes 2024-04-13T13:32:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2536117248 bytes 2024-04-13T13:32:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2588903424 bytes 2024-04-13T13:32:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2648869888 bytes 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_44 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_44 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_44 128M 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_44 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2024-04-13T13:32:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:32:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-04-13T13:32:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_44 2024-04-13T13:32:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-04-13T13:32:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:32:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_44': 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 142860d891d9 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.142860d891d9 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:32:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:32:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:30 2024 2024-04-13T13:32:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:30 2024 2024-04-13T13:32:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:30 2024 2024-04-13T13:32:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:32:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:32:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8f12f1ec-dc11-465e-a774-65eca20bf996 2024-04-13T13:32:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:32:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:32:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_44 2024-04-13T13:32:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_44 2024-04-13T13:32:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:32:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=14 2024-04-13T13:32:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:32:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:32:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 81828864 bytes 2024-04-13T13:32:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 82440192 bytes 2024-04-13T13:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 105365504 bytes 2024-04-13T13:32:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 208370688 bytes 2024-04-13T13:32:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 243522560 bytes 2024-04-13T13:32:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 255837184 bytes 2024-04-13T13:32:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 262158336 bytes 2024-04-13T13:32:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 318228480 bytes 2024-04-13T13:32:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 367779840 bytes 2024-04-13T13:32:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 427724800 bytes 2024-04-13T13:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 483125248 bytes 2024-04-13T13:32:33.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 558774272 bytes 2024-04-13T13:32:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 559392768 bytes 2024-04-13T13:32:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 567640064 bytes 2024-04-13T13:32:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 636493824 bytes 2024-04-13T13:32:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 649887744 bytes 2024-04-13T13:32:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 741947392 bytes 2024-04-13T13:32:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 772130816 bytes 2024-04-13T13:32:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 789050368 bytes 2024-04-13T13:32:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 789682176 bytes 2024-04-13T13:32:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 853562368 bytes 2024-04-13T13:32:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 881282048 bytes 2024-04-13T13:32:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 916408320 bytes 2024-04-13T13:32:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 921434112 bytes 2024-04-13T13:32:35.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 960804864 bytes 2024-04-13T13:32:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 961439744 bytes 2024-04-13T13:32:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1029690368 bytes 2024-04-13T13:32:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1082600448 bytes 2024-04-13T13:32:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1133982720 bytes 2024-04-13T13:32:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1176546304 bytes 2024-04-13T13:32:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1299042304 bytes 2024-04-13T13:32:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1332420608 bytes 2024-04-13T13:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1395395584 bytes 2024-04-13T13:32:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1415819264 bytes 2024-04-13T13:32:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1460591616 bytes 2024-04-13T13:32:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1495550976 bytes 2024-04-13T13:32:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1519537152 bytes 2024-04-13T13:32:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1604957184 bytes 2024-04-13T13:32:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1629346816 bytes 2024-04-13T13:32:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1710436352 bytes 2024-04-13T13:32:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1740025856 bytes 2024-04-13T13:32:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1786302464 bytes 2024-04-13T13:32:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1799610368 bytes 2024-04-13T13:32:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1828127744 bytes 2024-04-13T13:32:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1857995776 bytes 2024-04-13T13:32:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1945467904 bytes 2024-04-13T13:32:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1971766272 bytes 2024-04-13T13:32:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2016624640 bytes 2024-04-13T13:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2035130368 bytes 2024-04-13T13:32:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2093502464 bytes 2024-04-13T13:32:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2167337984 bytes 2024-04-13T13:32:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2187821056 bytes 2024-04-13T13:32:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2252740608 bytes 2024-04-13T13:32:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2290364416 bytes 2024-04-13T13:32:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2318678016 bytes 2024-04-13T13:32:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2334866432 bytes 2024-04-13T13:32:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2349468672 bytes 2024-04-13T13:32:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2365427712 bytes 2024-04-13T13:32:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2420030464 bytes 2024-04-13T13:32:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2462131200 bytes 2024-04-13T13:32:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2500276224 bytes 2024-04-13T13:32:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2539741184 bytes 2024-04-13T13:32:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2564105216 bytes 2024-04-13T13:32:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2620046336 bytes 2024-04-13T13:32:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2719938560 bytes 2024-04-13T13:32:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2740400128 bytes 2024-04-13T13:32:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2767168512 bytes 2024-04-13T13:32:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2792264704 bytes 2024-04-13T13:32:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2792870912 bytes 2024-04-13T13:32:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2815389696 bytes 2024-04-13T13:32:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2850473984 bytes 2024-04-13T13:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2904520704 bytes 2024-04-13T13:32:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2935029760 bytes 2024-04-13T13:32:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 2978159616 bytes 2024-04-13T13:32:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 3026430976 bytes 2024-04-13T13:32:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3037089792 bytes 2024-04-13T13:32:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 3062578176 bytes 2024-04-13T13:32:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7800 IOs, 3084544000 bytes 2024-04-13T13:32:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7900 IOs, 3219958784 bytes 2024-04-13T13:32:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:32:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:32:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:32:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:32:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:32:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_45 2024-04-13T13:32:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_45 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_45 128M 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_45 2024-04-13T13:32:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:32:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:32:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:32:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2024-04-13T13:32:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:32:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-04-13T13:32:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_45 2024-04-13T13:32:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-04-13T13:32:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:32:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_45': 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 143189c6657 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.143189c6657 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:32:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:32:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:32:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:45 2024 2024-04-13T13:32:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:45 2024 2024-04-13T13:32:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:45 2024 2024-04-13T13:32:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:32:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:32:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 4722198d-f141-466b-aec9-9f1bd91cfc3f 2024-04-13T13:32:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:32:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:32:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_45 2024-04-13T13:32:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_45 2024-04-13T13:32:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:32:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=5 2024-04-13T13:32:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:32:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:32:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 62846976 bytes 2024-04-13T13:32:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 102915072 bytes 2024-04-13T13:32:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 131785728 bytes 2024-04-13T13:32:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 138911744 bytes 2024-04-13T13:32:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 187052032 bytes 2024-04-13T13:32:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 263888896 bytes 2024-04-13T13:32:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 313104384 bytes 2024-04-13T13:32:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 373521408 bytes 2024-04-13T13:32:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 431460352 bytes 2024-04-13T13:32:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 468243456 bytes 2024-04-13T13:32:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 478523392 bytes 2024-04-13T13:32:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 520816640 bytes 2024-04-13T13:32:48.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 578091008 bytes 2024-04-13T13:32:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 611414016 bytes 2024-04-13T13:32:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 669767680 bytes 2024-04-13T13:32:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 694353920 bytes 2024-04-13T13:32:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 782740480 bytes 2024-04-13T13:32:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 801688576 bytes 2024-04-13T13:32:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 865263616 bytes 2024-04-13T13:32:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 967532544 bytes 2024-04-13T13:32:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 996085760 bytes 2024-04-13T13:32:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1061266432 bytes 2024-04-13T13:32:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1061869568 bytes 2024-04-13T13:32:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1080579072 bytes 2024-04-13T13:32:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1110702080 bytes 2024-04-13T13:32:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1111313408 bytes 2024-04-13T13:32:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:32:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:32:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:32:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:32:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:32:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_46 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_46 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_46 128M 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_46 2024-04-13T13:32:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:32:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:32:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:32:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:32:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2024-04-13T13:32:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:32:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-04-13T13:32:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_46 2024-04-13T13:32:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:32:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-04-13T13:32:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:32:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-04-13T13:32:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_46': 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 143b2fda7678 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.143b2fda7678 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:50 2024 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:50 2024 2024-04-13T13:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:50 2024 2024-04-13T13:32:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:32:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:32:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: fe7f2378-9fe8-4c98-84bb-5958971e249f 2024-04-13T13:32:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:32:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:32:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_46 2024-04-13T13:32:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:32:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:32:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_46 2024-04-13T13:32:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:32:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=12 2024-04-13T13:32:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:32:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:32:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 87397376 bytes 2024-04-13T13:32:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 128880640 bytes 2024-04-13T13:32:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 143482880 bytes 2024-04-13T13:32:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 190931968 bytes 2024-04-13T13:32:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 267100160 bytes 2024-04-13T13:32:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 315856896 bytes 2024-04-13T13:32:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 352549888 bytes 2024-04-13T13:32:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 394592256 bytes 2024-04-13T13:32:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 437262336 bytes 2024-04-13T13:32:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 479166464 bytes 2024-04-13T13:32:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 513480704 bytes 2024-04-13T13:32:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 539578368 bytes 2024-04-13T13:32:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 587942912 bytes 2024-04-13T13:32:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 699315200 bytes 2024-04-13T13:32:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 737027072 bytes 2024-04-13T13:32:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 758291456 bytes 2024-04-13T13:32:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 812612608 bytes 2024-04-13T13:32:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 835801088 bytes 2024-04-13T13:32:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 882217984 bytes 2024-04-13T13:32:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 918940672 bytes 2024-04-13T13:32:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 962466816 bytes 2024-04-13T13:32:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1034184704 bytes 2024-04-13T13:32:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1115862016 bytes 2024-04-13T13:32:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1158185984 bytes 2024-04-13T13:32:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1167818752 bytes 2024-04-13T13:32:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1182561280 bytes 2024-04-13T13:32:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1202007040 bytes 2024-04-13T13:32:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1307126784 bytes 2024-04-13T13:32:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1313992704 bytes 2024-04-13T13:32:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1371111424 bytes 2024-04-13T13:32:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1428770816 bytes 2024-04-13T13:32:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1525804032 bytes 2024-04-13T13:32:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1590505472 bytes 2024-04-13T13:32:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1631429632 bytes 2024-04-13T13:32:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1705227264 bytes 2024-04-13T13:32:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1724483584 bytes 2024-04-13T13:32:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1744261120 bytes 2024-04-13T13:32:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1842115584 bytes 2024-04-13T13:32:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1891138560 bytes 2024-04-13T13:32:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1898386432 bytes 2024-04-13T13:32:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1927404544 bytes 2024-04-13T13:32:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1949066240 bytes 2024-04-13T13:32:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1991135232 bytes 2024-04-13T13:32:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 2027047936 bytes 2024-04-13T13:33:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 2051343360 bytes 2024-04-13T13:33:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 2140670976 bytes 2024-04-13T13:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 2187704320 bytes 2024-04-13T13:33:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 2205004800 bytes 2024-04-13T13:33:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2205601792 bytes 2024-04-13T13:33:00.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2218698752 bytes 2024-04-13T13:33:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2282356736 bytes 2024-04-13T13:33:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2305992704 bytes 2024-04-13T13:33:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2361051136 bytes 2024-04-13T13:33:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2394927104 bytes 2024-04-13T13:33:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2395547648 bytes 2024-04-13T13:33:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2408749056 bytes 2024-04-13T13:33:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2468571136 bytes 2024-04-13T13:33:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2495566848 bytes 2024-04-13T13:33:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2544669696 bytes 2024-04-13T13:33:02.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2582255616 bytes 2024-04-13T13:33:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2680229888 bytes 2024-04-13T13:33:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2701323264 bytes 2024-04-13T13:33:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2732200960 bytes 2024-04-13T13:33:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2823070720 bytes 2024-04-13T13:33:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2897926144 bytes 2024-04-13T13:33:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:33:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:33:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:33:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:33:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_47 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_47 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_47 128M 2024-04-13T13:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_47 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2024-04-13T13:33:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:33:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-04-13T13:33:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_47 2024-04-13T13:33:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:33:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-04-13T13:33:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:33:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-04-13T13:33:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_47': 2024-04-13T13:33:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:33:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:33:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:33:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 14435dda3d9 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.14435dda3d9 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:33:03 2024 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:33:03 2024 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:33:03 2024 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1e77a3c2-50cb-4ed3-be13-ff3d35f16c46 2024-04-13T13:33:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:33:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:33:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_47 2024-04-13T13:33:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_47 2024-04-13T13:33:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:33:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=7 2024-04-13T13:33:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:33:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:33:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 60480512 bytes 2024-04-13T13:33:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 132415488 bytes 2024-04-13T13:33:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 150034432 bytes 2024-04-13T13:33:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 206342144 bytes 2024-04-13T13:33:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 234189824 bytes 2024-04-13T13:33:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 259201024 bytes 2024-04-13T13:33:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 280723456 bytes 2024-04-13T13:33:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 284681216 bytes 2024-04-13T13:33:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 326373376 bytes 2024-04-13T13:33:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 339303424 bytes 2024-04-13T13:33:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 344513536 bytes 2024-04-13T13:33:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 382206976 bytes 2024-04-13T13:33:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 407150592 bytes 2024-04-13T13:33:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 428743680 bytes 2024-04-13T13:33:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 484168704 bytes 2024-04-13T13:33:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 491375616 bytes 2024-04-13T13:33:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 524642304 bytes 2024-04-13T13:33:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 577150976 bytes 2024-04-13T13:33:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 636513280 bytes 2024-04-13T13:33:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 676559872 bytes 2024-04-13T13:33:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 699239424 bytes 2024-04-13T13:33:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 752204800 bytes 2024-04-13T13:33:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 771683328 bytes 2024-04-13T13:33:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 813987840 bytes 2024-04-13T13:33:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 871757824 bytes 2024-04-13T13:33:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 928102400 bytes 2024-04-13T13:33:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 951211008 bytes 2024-04-13T13:33:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1009838080 bytes 2024-04-13T13:33:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1035450368 bytes 2024-04-13T13:33:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1053264896 bytes 2024-04-13T13:33:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1125445632 bytes 2024-04-13T13:33:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1172196352 bytes 2024-04-13T13:33:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1236958208 bytes 2024-04-13T13:33:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1306658816 bytes 2024-04-13T13:33:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1362141184 bytes 2024-04-13T13:33:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1407869952 bytes 2024-04-13T13:33:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1490708480 bytes 2024-04-13T13:33:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1533144064 bytes 2024-04-13T13:33:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1561443328 bytes 2024-04-13T13:33:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:33:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:33:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:33:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:33:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_48 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_48 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_48 128M 2024-04-13T13:33:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_48 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2024-04-13T13:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-04-13T13:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_48 2024-04-13T13:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-04-13T13:33:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:33:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-04-13T13:33:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_48': 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 144c743099bd 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.144c743099bd 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:33:11 2024 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:33:11 2024 2024-04-13T13:33:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:33:11 2024 2024-04-13T13:33:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:33:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:33:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: fee9868f-6c82-4814-a3f0-2803aaff14e9 2024-04-13T13:33:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:33:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:33:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_48 2024-04-13T13:33:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_48 2024-04-13T13:33:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:33:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=13 2024-04-13T13:33:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:33:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:33:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 56237056 bytes 2024-04-13T13:33:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 81597440 bytes 2024-04-13T13:33:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 165003264 bytes 2024-04-13T13:33:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 211510272 bytes 2024-04-13T13:33:13.824 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6177593273850536) > weight (0.25) 2024-04-13T13:33:13.824 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-13T13:33:13.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-13T13:33:13.824 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-13T13:33:13.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:2024-04-13T13:33:13.833+0000 7f9a6c5f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 51424) UID: 1000 2024-04-13T13:33:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 268171264 bytes 2024-04-13T13:33:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 336344064 bytes 2024-04-13T13:33:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 343105536 bytes 2024-04-13T13:33:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 359787520 bytes 2024-04-13T13:33:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 389253120 bytes 2024-04-13T13:33:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 441532416 bytes 2024-04-13T13:33:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 520151040 bytes 2024-04-13T13:33:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 524130304 bytes 2024-04-13T13:33:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 529575936 bytes 2024-04-13T13:33:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 556429312 bytes 2024-04-13T13:33:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 576159744 bytes 2024-04-13T13:33:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 638221312 bytes 2024-04-13T13:33:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 689208320 bytes 2024-04-13T13:33:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 730272768 bytes 2024-04-13T13:33:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 772330496 bytes 2024-04-13T13:33:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 893886464 bytes 2024-04-13T13:33:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 971328512 bytes 2024-04-13T13:33:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 1015459840 bytes 2024-04-13T13:33:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 1043420160 bytes 2024-04-13T13:33:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 1059278848 bytes 2024-04-13T13:33:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 1119129600 bytes 2024-04-13T13:33:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 1119742976 bytes 2024-04-13T13:33:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 1194715136 bytes 2024-04-13T13:33:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 1280586752 bytes 2024-04-13T13:33:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 1340550144 bytes 2024-04-13T13:33:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1393778688 bytes 2024-04-13T13:33:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1401176064 bytes 2024-04-13T13:33:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1438531584 bytes 2024-04-13T13:33:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1505105920 bytes 2024-04-13T13:33:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1551726592 bytes 2024-04-13T13:33:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1615224832 bytes 2024-04-13T13:33:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1640839168 bytes 2024-04-13T13:33:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1663682560 bytes 2024-04-13T13:33:19.825 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-13T13:33:19.825 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:33:19.826 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-13T13:33:19.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-13T13:33:19.826 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-13T13:33:19.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-13T13:33:19.828 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-04-13T13:33:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1707383808 bytes 2024-04-13T13:33:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1724283904 bytes 2024-04-13T13:33:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1760377856 bytes 2024-04-13T13:33:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1819399168 bytes 2024-04-13T13:33:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1843826688 bytes 2024-04-13T13:33:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1859880960 bytes 2024-04-13T13:33:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1873954816 bytes 2024-04-13T13:33:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1895855104 bytes 2024-04-13T13:33:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1947889664 bytes 2024-04-13T13:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1952625664 bytes 2024-04-13T13:33:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1954680832 bytes 2024-04-13T13:33:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 2006000640 bytes 2024-04-13T13:33:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 2023470080 bytes 2024-04-13T13:33:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 2029675520 bytes 2024-04-13T13:33:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2089351168 bytes 2024-04-13T13:33:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2163927040 bytes 2024-04-13T13:33:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2201500672 bytes 2024-04-13T13:33:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2223271936 bytes 2024-04-13T13:33:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2225845248 bytes 2024-04-13T13:33:22.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2264668160 bytes 2024-04-13T13:33:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2326207488 bytes 2024-04-13T13:33:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2346003456 bytes 2024-04-13T13:33:23.156 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:33:23.162+0000 7fd27fa27640 -1 bdev(0x55cbc596ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1187-gd540ebac/rpm/el9/BUILD/ceph-18.2.2-1187-gd540ebac/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fd27fa27640 time 2024-04-13T13:33:23.164866+0000 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1187-gd540ebac/rpm/el9/BUILD/ceph-18.2.2-1187-gd540ebac/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55cbc3130962] 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55cbc3ac8f60] 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 3: ceph-osd(+0xd620d1) [0x55cbc3ac90d1] 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7fd28d29f802] 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7fd28d23f450] 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:33:23.163+0000 7fd27fa27640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1187-gd540ebac/rpm/el9/BUILD/ceph-18.2.2-1187-gd540ebac/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fd27fa27640 time 2024-04-13T13:33:23.164866+0000 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1187-gd540ebac/rpm/el9/BUILD/ceph-18.2.2-1187-gd540ebac/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-13T13:33:23.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55cbc3130962] 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55cbc3ac8f60] 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 3: ceph-osd(+0xd620d1) [0x55cbc3ac90d1] 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7fd28d29f802] 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7fd28d23f450] 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:*** Caught signal (Aborted) ** 2024-04-13T13:33:23.158 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: in thread 7fd27fa27640 thread_name:bstore_aio 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fd28d254db0] 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fd28d2a154c] 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 3: raise() 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 4: abort() 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55cbc3130a23] 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55cbc3ac8f60] 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 7: ceph-osd(+0xd620d1) [0x55cbc3ac90d1] 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fd28d29f802] 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fd28d23f450] 2024-04-13T13:33:23.159 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2024-04-13T13:33:23.164+0000 7fd27fa27640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: in thread 7fd27fa27640 thread_name:bstore_aio 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fd28d254db0] 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fd28d2a154c] 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 3: raise() 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 4: abort() 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55cbc3130a23] 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55cbc3ac8f60] 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 7: ceph-osd(+0xd620d1) [0x55cbc3ac90d1] 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fd28d29f802] 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fd28d23f450] 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:33:23.160 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.172 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: -14> 2024-04-13T13:33:23.162+0000 7fd27fa27640 -1 bdev(0x55cbc596ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-04-13T13:33:23.172 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: -5> 2024-04-13T13:33:23.163+0000 7fd27fa27640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1187-gd540ebac/rpm/el9/BUILD/ceph-18.2.2-1187-gd540ebac/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fd27fa27640 time 2024-04-13T13:33:23.164866+0000 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1187-gd540ebac/rpm/el9/BUILD/ceph-18.2.2-1187-gd540ebac/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55cbc3130962] 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55cbc3ac8f60] 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 3: ceph-osd(+0xd620d1) [0x55cbc3ac90d1] 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7fd28d29f802] 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7fd28d23f450] 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 0> 2024-04-13T13:33:23.164+0000 7fd27fa27640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: in thread 7fd27fa27640 thread_name:bstore_aio 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:33:23.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fd28d254db0] 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fd28d2a154c] 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 3: raise() 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 4: abort() 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55cbc3130a23] 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55cbc3ac8f60] 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 7: ceph-osd(+0xd620d1) [0x55cbc3ac90d1] 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fd28d29f802] 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fd28d23f450] 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:33:23.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.181 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: -329> 2024-04-13T13:33:23.162+0000 7fd27fa27640 -1 bdev(0x55cbc596ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-04-13T13:33:23.181 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: -320> 2024-04-13T13:33:23.163+0000 7fd27fa27640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1187-gd540ebac/rpm/el9/BUILD/ceph-18.2.2-1187-gd540ebac/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fd27fa27640 time 2024-04-13T13:33:23.164866+0000 2024-04-13T13:33:23.181 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1187-gd540ebac/rpm/el9/BUILD/ceph-18.2.2-1187-gd540ebac/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55cbc3130962] 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55cbc3ac8f60] 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 3: ceph-osd(+0xd620d1) [0x55cbc3ac90d1] 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7fd28d29f802] 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7fd28d23f450] 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: -315> 2024-04-13T13:33:23.164+0000 7fd27fa27640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: in thread 7fd27fa27640 thread_name:bstore_aio 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:33:23.182 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fd28d254db0] 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fd28d2a154c] 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 3: raise() 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 4: abort() 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55cbc3130a23] 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55cbc3ac8f60] 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 7: ceph-osd(+0xd620d1) [0x55cbc3ac90d1] 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fd28d29f802] 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fd28d23f450] 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:33:23.183 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr: 2024-04-13T13:33:23.271 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:daemon-helper: command failed with exit status 5 2024-04-13T13:33:24.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-04-13T13:33:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2404288512 bytes 2024-04-13T13:33:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2444264448 bytes 2024-04-13T13:33:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2534064128 bytes 2024-04-13T13:33:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2553483264 bytes 2024-04-13T13:33:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2616914944 bytes 2024-04-13T13:33:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2658762752 bytes 2024-04-13T13:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2691408896 bytes 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_49 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_49 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_49 128M 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_49 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:33:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:33:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2024-04-13T13:33:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:33:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-04-13T13:33:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_49 2024-04-13T13:33:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:33:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-04-13T13:33:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:33:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_49': 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 145b552fe9c 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.145b552fe9c 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:33:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:33:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:33:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:33:25 2024 2024-04-13T13:33:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:33:25 2024 2024-04-13T13:33:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:33:25 2024 2024-04-13T13:33:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:33:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:33:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 89d28a1c-d65d-4fb9-b8cc-35062079a2b1 2024-04-13T13:33:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:33:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:33:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_49 2024-04-13T13:33:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_49 2024-04-13T13:33:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:33:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=10 2024-04-13T13:33:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:33:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:33:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 17583104 bytes 2024-04-13T13:33:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 29620224 bytes 2024-04-13T13:33:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 113179648 bytes 2024-04-13T13:33:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 131941376 bytes 2024-04-13T13:33:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 155455488 bytes 2024-04-13T13:33:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 183803904 bytes 2024-04-13T13:33:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 217224192 bytes 2024-04-13T13:33:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 298915840 bytes 2024-04-13T13:33:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 366342144 bytes 2024-04-13T13:33:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 387069952 bytes 2024-04-13T13:33:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 399092736 bytes 2024-04-13T13:33:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 413908992 bytes 2024-04-13T13:33:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 424840192 bytes 2024-04-13T13:33:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 486013952 bytes 2024-04-13T13:33:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 576401408 bytes 2024-04-13T13:33:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 611038208 bytes 2024-04-13T13:33:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 626124800 bytes 2024-04-13T13:33:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 657898496 bytes 2024-04-13T13:33:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 680897536 bytes 2024-04-13T13:33:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 760940544 bytes 2024-04-13T13:33:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 774697984 bytes 2024-04-13T13:33:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 793412608 bytes 2024-04-13T13:33:29.728 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9770111549631812) > weight (0.25) 2024-04-13T13:33:29.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7498006087508894) > weight (0.25) 2024-04-13T13:33:29.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5286541912188597) > weight (0.25) 2024-04-13T13:33:29.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9983658632347893) > weight (0.25) 2024-04-13T13:33:29.729 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-04-13T13:33:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 855239680 bytes 2024-04-13T13:33:29.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2024-04-13T13:33:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 877608960 bytes 2024-04-13T13:33:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 879324160 bytes 2024-04-13T13:33:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 881348608 bytes 2024-04-13T13:33:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 908684288 bytes 2024-04-13T13:33:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 919506944 bytes 2024-04-13T13:33:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 960354304 bytes 2024-04-13T13:33:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1011266560 bytes 2024-04-13T13:33:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1057453056 bytes 2024-04-13T13:33:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1080652800 bytes 2024-04-13T13:33:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1081266176 bytes 2024-04-13T13:33:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1106580480 bytes 2024-04-13T13:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1122745344 bytes 2024-04-13T13:33:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1149698048 bytes 2024-04-13T13:33:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1220096000 bytes 2024-04-13T13:33:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1241554944 bytes 2024-04-13T13:33:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1310184448 bytes 2024-04-13T13:33:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1351372800 bytes 2024-04-13T13:33:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1424651264 bytes 2024-04-13T13:33:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1438047232 bytes 2024-04-13T13:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1493270528 bytes 2024-04-13T13:33:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1564220416 bytes 2024-04-13T13:33:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1638738944 bytes 2024-04-13T13:33:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1662652416 bytes 2024-04-13T13:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1725454336 bytes 2024-04-13T13:33:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1776068608 bytes 2024-04-13T13:33:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1802005504 bytes 2024-04-13T13:33:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1851488256 bytes 2024-04-13T13:33:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1886357504 bytes 2024-04-13T13:33:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 1932873728 bytes 2024-04-13T13:33:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 1975931904 bytes 2024-04-13T13:33:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 1988555776 bytes 2024-04-13T13:33:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2005600256 bytes 2024-04-13T13:33:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2060732416 bytes 2024-04-13T13:33:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2080600064 bytes 2024-04-13T13:33:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2118780928 bytes 2024-04-13T13:33:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2160336896 bytes 2024-04-13T13:33:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2261938176 bytes 2024-04-13T13:33:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2285489152 bytes 2024-04-13T13:33:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2308215808 bytes 2024-04-13T13:33:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2343335936 bytes 2024-04-13T13:33:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2398748672 bytes 2024-04-13T13:33:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2484804608 bytes 2024-04-13T13:33:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2531349504 bytes 2024-04-13T13:33:35.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2024-04-13T13:33:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2559007744 bytes 2024-04-13T13:33:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2621950976 bytes 2024-04-13T13:33:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2668407808 bytes 2024-04-13T13:33:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2725089280 bytes 2024-04-13T13:33:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2732125184 bytes 2024-04-13T13:33:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2826767360 bytes 2024-04-13T13:33:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2947147776 bytes 2024-04-13T13:33:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 2950818816 bytes 2024-04-13T13:33:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 2999421952 bytes 2024-04-13T13:33:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3080326144 bytes 2024-04-13T13:33:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 3108813824 bytes 2024-04-13T13:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_50 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_50 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n snapshot ']' 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror image_50 128M 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_50 2024-04-13T13:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2024-04-13T13:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n 128M ']' 2024-04-13T13:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128M 2024-04-13T13:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2024-04-13T13:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2024-04-13T13:33:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' image = image ']' 2024-04-13T13:33:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-04-13T13:33:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_50 2024-04-13T13:33:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=snapshot 2024-04-13T13:33:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-04-13T13:33:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2024-04-13T13:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-04-13T13:33:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_50': 2024-04-13T13:33:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:33:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:33:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:33:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1467bb372fb0 2024-04-13T13:33:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1467bb372fb0 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:33:36 2024 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:33:36 2024 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:33:36 2024 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: bb8a902e-d932-412f-bc77-def5842a6663 2024-04-13T13:33:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2024-04-13T13:33:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -n '' ']' 2024-04-13T13:33:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stress_write_image cluster2 mirror image_50 2024-04-13T13:33:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_50 2024-04-13T13:33:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-13T13:33:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local duration=13 2024-04-13T13:33:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:33:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-04-13T13:33:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:100 IOs, 41172992 bytes 2024-04-13T13:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:200 IOs, 114962432 bytes 2024-04-13T13:33:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:300 IOs, 160191488 bytes 2024-04-13T13:33:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:400 IOs, 191570944 bytes 2024-04-13T13:33:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:500 IOs, 249588736 bytes 2024-04-13T13:33:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:600 IOs, 303887360 bytes 2024-04-13T13:33:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:700 IOs, 314229760 bytes 2024-04-13T13:33:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:800 IOs, 336874496 bytes 2024-04-13T13:33:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:900 IOs, 392527872 bytes 2024-04-13T13:33:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1000 IOs, 406203392 bytes 2024-04-13T13:33:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1100 IOs, 429224960 bytes 2024-04-13T13:33:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1200 IOs, 429828096 bytes 2024-04-13T13:33:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1300 IOs, 434061312 bytes 2024-04-13T13:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1400 IOs, 456588288 bytes 2024-04-13T13:33:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1500 IOs, 494918656 bytes 2024-04-13T13:33:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1600 IOs, 495551488 bytes 2024-04-13T13:33:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1700 IOs, 576950272 bytes 2024-04-13T13:33:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1800 IOs, 627615744 bytes 2024-04-13T13:33:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:1900 IOs, 679065600 bytes 2024-04-13T13:33:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2000 IOs, 720704512 bytes 2024-04-13T13:33:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2100 IOs, 779163648 bytes 2024-04-13T13:33:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2200 IOs, 790899712 bytes 2024-04-13T13:33:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2300 IOs, 884555776 bytes 2024-04-13T13:33:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2400 IOs, 906438656 bytes 2024-04-13T13:33:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2500 IOs, 923335680 bytes 2024-04-13T13:33:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2600 IOs, 947859456 bytes 2024-04-13T13:33:40.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2024-04-13T13:33:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2700 IOs, 956197888 bytes 2024-04-13T13:33:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2800 IOs, 985810944 bytes 2024-04-13T13:33:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:2900 IOs, 986589184 bytes 2024-04-13T13:33:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3000 IOs, 1021744128 bytes 2024-04-13T13:33:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3100 IOs, 1022340096 bytes 2024-04-13T13:33:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3200 IOs, 1121249280 bytes 2024-04-13T13:33:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3300 IOs, 1133333504 bytes 2024-04-13T13:33:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3400 IOs, 1211549696 bytes 2024-04-13T13:33:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3500 IOs, 1222006784 bytes 2024-04-13T13:33:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3600 IOs, 1285991424 bytes 2024-04-13T13:33:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3700 IOs, 1341636608 bytes 2024-04-13T13:33:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3800 IOs, 1374025728 bytes 2024-04-13T13:33:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:3900 IOs, 1429738496 bytes 2024-04-13T13:33:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4000 IOs, 1502491648 bytes 2024-04-13T13:33:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4100 IOs, 1566632960 bytes 2024-04-13T13:33:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4200 IOs, 1583788032 bytes 2024-04-13T13:33:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4300 IOs, 1625918464 bytes 2024-04-13T13:33:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4400 IOs, 1689555968 bytes 2024-04-13T13:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4500 IOs, 1772353536 bytes 2024-04-13T13:33:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4600 IOs, 1799670784 bytes 2024-04-13T13:33:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4700 IOs, 1881635840 bytes 2024-04-13T13:33:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4800 IOs, 1887899648 bytes 2024-04-13T13:33:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:4900 IOs, 1888504832 bytes 2024-04-13T13:33:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5000 IOs, 1949700096 bytes 2024-04-13T13:33:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5100 IOs, 1967237120 bytes 2024-04-13T13:33:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5200 IOs, 2007483392 bytes 2024-04-13T13:33:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5300 IOs, 2059083776 bytes 2024-04-13T13:33:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5400 IOs, 2119078912 bytes 2024-04-13T13:33:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5500 IOs, 2183398400 bytes 2024-04-13T13:33:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5600 IOs, 2234612736 bytes 2024-04-13T13:33:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5700 IOs, 2287184896 bytes 2024-04-13T13:33:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5800 IOs, 2308777984 bytes 2024-04-13T13:33:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:5900 IOs, 2337675264 bytes 2024-04-13T13:33:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6000 IOs, 2362286080 bytes 2024-04-13T13:33:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6100 IOs, 2381915136 bytes 2024-04-13T13:33:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6200 IOs, 2436639744 bytes 2024-04-13T13:33:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6300 IOs, 2491539456 bytes 2024-04-13T13:33:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6400 IOs, 2564998144 bytes 2024-04-13T13:33:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6500 IOs, 2584273920 bytes 2024-04-13T13:33:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6600 IOs, 2629989376 bytes 2024-04-13T13:33:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6700 IOs, 2713311232 bytes 2024-04-13T13:33:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6800 IOs, 2713950208 bytes 2024-04-13T13:33:46.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2024-04-13T13:33:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:6900 IOs, 2739770368 bytes 2024-04-13T13:33:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7000 IOs, 2765161472 bytes 2024-04-13T13:33:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7100 IOs, 2806295552 bytes 2024-04-13T13:33:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7200 IOs, 2900812800 bytes 2024-04-13T13:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7300 IOs, 2914612224 bytes 2024-04-13T13:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7400 IOs, 2947836928 bytes 2024-04-13T13:33:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7500 IOs, 2974316544 bytes 2024-04-13T13:33:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7600 IOs, 3037291520 bytes 2024-04-13T13:33:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7700 IOs, 3112792064 bytes 2024-04-13T13:33:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7800 IOs, 3143985152 bytes 2024-04-13T13:33:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:7900 IOs, 3190809600 bytes 2024-04-13T13:33:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8000 IOs, 3238302720 bytes 2024-04-13T13:33:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8100 IOs, 3261239296 bytes 2024-04-13T13:33:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8200 IOs, 3279378432 bytes 2024-04-13T13:33:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8300 IOs, 3329361920 bytes 2024-04-13T13:33:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8400 IOs, 3341102080 bytes 2024-04-13T13:33:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8500 IOs, 3353431040 bytes 2024-04-13T13:33:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8600 IOs, 3389131776 bytes 2024-04-13T13:33:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8700 IOs, 3428063232 bytes 2024-04-13T13:33:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8800 IOs, 3492718592 bytes 2024-04-13T13:33:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:8900 IOs, 3549492224 bytes 2024-04-13T13:33:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9000 IOs, 3611597824 bytes 2024-04-13T13:33:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9100 IOs, 3653409792 bytes 2024-04-13T13:33:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9200 IOs, 3686443008 bytes 2024-04-13T13:33:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9300 IOs, 3706182656 bytes 2024-04-13T13:33:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9400 IOs, 3726651392 bytes 2024-04-13T13:33:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9500 IOs, 3771927552 bytes 2024-04-13T13:33:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9600 IOs, 3818262528 bytes 2024-04-13T13:33:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9700 IOs, 3834874880 bytes 2024-04-13T13:33:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9800 IOs, 3870829568 bytes 2024-04-13T13:33:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:9900 IOs, 3877479424 bytes 2024-04-13T13:33:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:10000 IOs, 3900076032 bytes 2024-04-13T13:33:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:10100 IOs, 3954111488 bytes 2024-04-13T13:33:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:10200 IOs, 4000802816 bytes 2024-04-13T13:33:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:10300 IOs, 4011233280 bytes 2024-04-13T13:33:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:10400 IOs, 4045666304 bytes 2024-04-13T13:33:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:10500 IOs, 4081142784 bytes 2024-04-13T13:33:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=124 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 124 -eq 124 ']' 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_count=50 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_count=0 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 40 2024-04-13T13:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in `seq 1 40` 2024-04-13T13:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 1 -ne 1 2024-04-13T13:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-13T13:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep 'images: ' 2024-04-13T13:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cut '-d ' -f 2 2024-04-13T13:33:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2024-04-13T13:33:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ count=50 2024-04-13T13:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 50 = 50 2024-04-13T13:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-04-13T13:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state 2024-04-13T13:33:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 40 2024-04-13T13:33:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in `seq 1 40` 2024-04-13T13:33:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 1 -ne 1 2024-04-13T13:33:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-13T13:33:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep 'image health:' 2024-04-13T13:33:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cut '-d ' -f 3 2024-04-13T13:33:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2024-04-13T13:33:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=OK 2024-04-13T13:33:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test OK = ERROR 2024-04-13T13:33:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test OK = OK 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_count=50 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_count=0 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 40 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in `seq 1 40` 2024-04-13T13:33:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 1 -ne 1 2024-04-13T13:33:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-13T13:33:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep 'images: ' 2024-04-13T13:33:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cut '-d ' -f 2 2024-04-13T13:33:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2024-04-13T13:33:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ count=50 2024-04-13T13:33:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 50 = 50 2024-04-13T13:33:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:33:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-04-13T13:33:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:33:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:33:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:33:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state 2024-04-13T13:33:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 40 2024-04-13T13:33:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in `seq 1 40` 2024-04-13T13:33:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 1 -ne 1 2024-04-13T13:33:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-13T13:33:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep 'image health:' 2024-04-13T13:33:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cut '-d ' -f 3 2024-04-13T13:33:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2024-04-13T13:33:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=WARNING 2024-04-13T13:33:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test WARNING = ERROR 2024-04-13T13:33:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test WARNING = OK 2024-04-13T13:33:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in `seq 1 40` 2024-04-13T13:33:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 2 -ne 1 2024-04-13T13:33:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 30 2024-04-13T13:33:51.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2024-04-13T13:33:56.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2024-04-13T13:34:02.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2024-04-13T13:34:07.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2024-04-13T13:34:13.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2024-04-13T13:34:18.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2024-04-13T13:34:19.829 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4663339487975704) > weight (0.25) 2024-04-13T13:34:19.830 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.47772104285651595) > weight (0.25) 2024-04-13T13:34:19.830 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-13T13:34:19.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-13T13:34:19.830 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-13T13:34:19.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi122.stderr:2024-04-13T13:34:19.839+0000 7fcf2e9f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 39633) UID: 1000 2024-04-13T13:34:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-13T13:34:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep 'image health:' 2024-04-13T13:34:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cut '-d ' -f 3 2024-04-13T13:34:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2024-04-13T13:34:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=OK 2024-04-13T13:34:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test OK = ERROR 2024-04-13T13:34:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test OK = OK 2024-04-13T13:34:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog 'TEST: compare many images' 2024-04-13T13:34:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.6pyizPAJcx/rbd-mirror.test.log 2024-04-13T13:34:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ date '+%F %T' 2024-04-13T13:34:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13 13:34:20 TEST: compare many images 2024-04-13T13:34:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13 13:34:20 TEST: compare many images 2024-04-13T13:34:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 1 50 2024-04-13T13:34:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:34:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_1 2024-04-13T13:34:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_1 snap 2024-04-13T13:34:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:34:20.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi122.stderr:2024-04-13T13:34:20.973+0000 7ff1a89f7640 -1 rbd::mirror::ImageReplayer: 0x5582117c8500 [3/41ea30c5-3df3-4041-8289-68ecbae98d63] register_admin_socket_hook: error registering admin socket commands 2024-04-13T13:34:20.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi122.stderr:2024-04-13T13:34:20.973+0000 7ff1a91f8640 -1 rbd::mirror::ImageReplayer: 0x5582117c8500 [3/41ea30c5-3df3-4041-8289-68ecbae98d63] register_admin_socket_hook: error registering admin socket commands 2024-04-13T13:34:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-04-13T13:34:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:34:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:34:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:34:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:34:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:34:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-04-13T13:34:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:34:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:34:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:34:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-04-13T13:34:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2024-04-13T13:34:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/image_15 2024-04-13T13:34:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:34:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:34:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:34:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:34:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-04-13T13:34:23.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2024-04-13T13:34:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:34:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:34:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:34:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:34:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:34:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:34:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:34:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-04-13T13:34:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_1.status 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_1.status 2024-04-13T13:34:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_1.status 2024-04-13T13:34:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_1.status 2024-04-13T13:34:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_1.status 2024-04-13T13:34:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-04-13T13:34:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-04-13T13:34:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 119 2024-04-13T13:34:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_1.status 2024-04-13T13:34:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_1.status 2024-04-13T13:34:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-04-13T13:34:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:34:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=119 2024-04-13T13:34:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:34:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-13T13:34:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=68 2024-04-13T13:34:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 119 = 68 2024-04-13T13:34:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:34:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-13T13:34:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=68 2024-04-13T13:34:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 119 = 68 2024-04-13T13:34:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:34:25.832 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-13T13:34:25.832 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:34:25.832 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-13T13:34:25.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-13T13:34:25.832 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-13T13:34:25.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-13T13:34:25.835 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-04-13T13:34:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-13T13:34:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:34:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:34:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:34:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:34:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 119 = '' 2024-04-13T13:34:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:34:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.status 2024-04-13T13:34:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-13T13:34:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=119 2024-04-13T13:34:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 119 = 119 2024-04-13T13:34:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-04-13T13:34:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:34:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-04-13T13:34:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_1': 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab9e4a96b9 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab9e4a96b9 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:13 2024 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:13 2024 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:13 2024 2024-04-13T13:34:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:34:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:34:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:34:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f 2024-04-13T13:34:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_1 snap 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_1 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_1.export 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.export 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_1.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.export 2024-04-13T13:34:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_1.export 2024-04-13T13:34:29.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2024-04-13T13:34:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:34:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.export 2024-04-13T13:34:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:34:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_1.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.export 2024-04-13T13:34:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_1.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_1.export 2024-04-13T13:34:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_2 2024-04-13T13:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_2 snap 2024-04-13T13:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:34:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-04-13T13:34:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-04-13T13:34:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:34:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-04-13T13:34:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:34:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:34:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:34:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-04-13T13:34:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:34:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:34:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:34:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:34:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:34:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:34:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:34:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:34:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:34:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:34:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-04-13T13:34:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 2 closest matches: 2024-04-13T13:34:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/image_20 2024-04-13T13:34:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/image_23 2024-04-13T13:34:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-04-13T13:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:34:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:34:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:34:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:34:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:34:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:34:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-04-13T13:34:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:34:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:34:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:34:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:34:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:34:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:34:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:34:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-04-13T13:34:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_2.status 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_2.status 2024-04-13T13:34:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_2.status 2024-04-13T13:34:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_2.status 2024-04-13T13:34:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_2.status 2024-04-13T13:34:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-04-13T13:34:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-04-13T13:34:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 121 2024-04-13T13:34:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_2.status 2024-04-13T13:34:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_2.status 2024-04-13T13:34:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-04-13T13:34:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:34:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=121 2024-04-13T13:34:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:34:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-13T13:34:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=69 2024-04-13T13:34:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 121 = 69 2024-04-13T13:34:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:34:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-13T13:34:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=69 2024-04-13T13:34:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 121 = 69 2024-04-13T13:34:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-13T13:34:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:34:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:34:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:34:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:34:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 121 = '' 2024-04-13T13:34:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:34:34.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2024-04-13T13:34:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.status 2024-04-13T13:34:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-13T13:34:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=121 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 121 = 121 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:34:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-04-13T13:34:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_2': 2024-04-13T13:34:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:34:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:34:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:34:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0d185c1 2024-04-13T13:34:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0d185c1 2024-04-13T13:34:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:25 2024 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:25 2024 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:25 2024 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 45670160-3031-494c-bc36-3e5d858b75c1 2024-04-13T13:34:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_2 snap 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_2 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_2.export 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.export 2024-04-13T13:34:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_2.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.export 2024-04-13T13:34:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_2.export 2024-04-13T13:34:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:34:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.export 2024-04-13T13:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_2.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.export 2024-04-13T13:34:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_2.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_2.export 2024-04-13T13:34:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:34:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_3 2024-04-13T13:34:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_3 snap 2024-04-13T13:34:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:34:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-04-13T13:34:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-04-13T13:34:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:34:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:34:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:34:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-04-13T13:34:40.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2024-04-13T13:34:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 3 closest matches: 2024-04-13T13:34:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/image_35 2024-04-13T13:34:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/image_36 2024-04-13T13:34:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/image_39 2024-04-13T13:34:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:34:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:34:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:34:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:34:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:34:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-04-13T13:34:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:34:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:34:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_3.status 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_3.status 2024-04-13T13:34:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_3.status 2024-04-13T13:34:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_3.status 2024-04-13T13:34:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_3.status 2024-04-13T13:34:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-04-13T13:34:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-04-13T13:34:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 123 2024-04-13T13:34:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_3.status 2024-04-13T13:34:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_3.status 2024-04-13T13:34:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-04-13T13:34:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:34:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=123 2024-04-13T13:34:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:34:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-13T13:34:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=70 2024-04-13T13:34:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 123 = 70 2024-04-13T13:34:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:34:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-13T13:34:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=70 2024-04-13T13:34:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 123 = 70 2024-04-13T13:34:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:34:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-13T13:34:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:34:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:34:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:34:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:34:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 123 = '' 2024-04-13T13:34:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-13T13:34:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:34:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:34:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:34:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:34:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 123 = '' 2024-04-13T13:34:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2024-04-13T13:34:45.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2024-04-13T13:34:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.status 2024-04-13T13:34:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-13T13:34:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=123 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 123 = 123 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:34:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_3': 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124cedb7a54f 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124cedb7a54f 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 173021c1-45fc-4ddf-ae55-e453164e52a8 2024-04-13T13:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_3 snap 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_3 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_3.export 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.export 2024-04-13T13:34:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_3.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.export 2024-04-13T13:34:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_3.export 2024-04-13T13:34:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:34:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.export 2024-04-13T13:34:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:34:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_3.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.export 2024-04-13T13:34:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_3.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_3.export 2024-04-13T13:34:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:34:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_4 2024-04-13T13:34:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_4 snap 2024-04-13T13:34:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:34:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-04-13T13:34:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:34:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:34:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:34:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:34:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:34:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-04-13T13:34:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:34:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:34:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:34:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:34:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:34:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-04-13T13:34:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:34:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:34:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:34:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:34:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:34:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_4.status 2024-04-13T13:34:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_4.status 2024-04-13T13:34:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_4.status 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_4.status 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_4.status 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-04-13T13:34:50.998 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2024-04-13T13:34:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 125 2024-04-13T13:34:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_4.status 2024-04-13T13:34:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_4.status 2024-04-13T13:34:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-04-13T13:34:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:34:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=125 2024-04-13T13:34:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:34:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-13T13:34:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=71 2024-04-13T13:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 125 = 71 2024-04-13T13:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:34:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-13T13:34:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:34:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:34:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:34:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:34:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 125 = '' 2024-04-13T13:34:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:34:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-13T13:34:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:34:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:34:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:34:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:34:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 125 = '' 2024-04-13T13:34:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:34:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.status 2024-04-13T13:34:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-13T13:34:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:34:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=125 2024-04-13T13:34:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 125 = 125 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:34:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-04-13T13:34:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_4': 2024-04-13T13:34:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:34:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:34:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:34:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b3fdd556cc 2024-04-13T13:34:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b3fdd556cc 2024-04-13T13:34:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8d69fb4b-4707-45d2-bd35-f5dc2bafa56b 2024-04-13T13:34:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_4 snap 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_4 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_4.export 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.export 2024-04-13T13:34:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_4.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.export 2024-04-13T13:34:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_4.export 2024-04-13T13:34:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:34:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.export 2024-04-13T13:34:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:34:56.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2024-04-13T13:34:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_4.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.export 2024-04-13T13:34:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_4.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_4.export 2024-04-13T13:34:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:34:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_5 2024-04-13T13:34:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_5 snap 2024-04-13T13:34:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:34:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:34:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:34:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-04-13T13:34:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:34:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:34:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:34:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:34:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:34:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:34:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:34:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:34:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:34:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:34:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-04-13T13:34:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-13T13:34:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:34:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:34:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:34:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-04-13T13:34:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:34:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:34:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:34:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:34:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-04-13T13:34:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:34:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:34:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:34:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:34:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:34:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:34:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_5.status 2024-04-13T13:34:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_5.status 2024-04-13T13:34:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_5.status 2024-04-13T13:34:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_5.status 2024-04-13T13:34:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_5.status 2024-04-13T13:34:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:34:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:34:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-04-13T13:34:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-04-13T13:34:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 127 2024-04-13T13:34:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_5.status 2024-04-13T13:34:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:34:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_5.status 2024-04-13T13:34:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-04-13T13:34:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:34:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:34:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=127 2024-04-13T13:34:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:34:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:34:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:34:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:34:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:34:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:34:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:34:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:34:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-13T13:34:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=72 2024-04-13T13:35:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 127 = 72 2024-04-13T13:35:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:35:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:35:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:35:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-13T13:35:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 127 = '' 2024-04-13T13:35:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:35:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:35:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:35:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-13T13:35:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 127 = '' 2024-04-13T13:35:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:35:01.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2024-04-13T13:35:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:35:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:35:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.status 2024-04-13T13:35:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-13T13:35:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=127 2024-04-13T13:35:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 127 = 127 2024-04-13T13:35:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-04-13T13:35:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:35:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_5': 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab185c36c 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab185c36c 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:58 2024 2024-04-13T13:35:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:25:58 2024 2024-04-13T13:35:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:58 2024 2024-04-13T13:35:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:35:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:35:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:35:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: b72ab703-5a35-4d41-90bc-db2c081ebc57 2024-04-13T13:35:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:35:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_5 snap 2024-04-13T13:35:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_5 2024-04-13T13:35:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:35:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_5.export 2024-04-13T13:35:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.export 2024-04-13T13:35:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_5.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.export 2024-04-13T13:35:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_5.export 2024-04-13T13:35:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.export 2024-04-13T13:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_5.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.export 2024-04-13T13:35:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_5.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_5.export 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_6 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_6 snap 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:35:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-04-13T13:35:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-04-13T13:35:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:35:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-04-13T13:35:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:35:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:35:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:35:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-04-13T13:35:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:35:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:35:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:35:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:35:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:35:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:35:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:35:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:35:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-04-13T13:35:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-13T13:35:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:35:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:35:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:35:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-04-13T13:35:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:35:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:35:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:35:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:35:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:35:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:35:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:35:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-04-13T13:35:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_6.status 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_6.status 2024-04-13T13:35:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_6.status 2024-04-13T13:35:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_6.status 2024-04-13T13:35:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_6.status 2024-04-13T13:35:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-04-13T13:35:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-04-13T13:35:07.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2024-04-13T13:35:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 129 2024-04-13T13:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_6.status 2024-04-13T13:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_6.status 2024-04-13T13:35:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-04-13T13:35:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:35:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=129 2024-04-13T13:35:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:35:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-13T13:35:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=73 2024-04-13T13:35:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 129 = 73 2024-04-13T13:35:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-13T13:35:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 129 = '' 2024-04-13T13:35:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-13T13:35:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 129 = '' 2024-04-13T13:35:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:35:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.status 2024-04-13T13:35:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-13T13:35:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=129 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 129 = 129 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-04-13T13:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_6': 2024-04-13T13:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0d7251c0a 2024-04-13T13:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0d7251c0a 2024-04-13T13:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:13 2024 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:13 2024 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:13 2024 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1466461b-b5d7-4bb3-9928-7a3a2fd5b22f 2024-04-13T13:35:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:35:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_6 snap 2024-04-13T13:35:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_6 2024-04-13T13:35:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:35:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_6.export 2024-04-13T13:35:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.export 2024-04-13T13:35:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_6.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.export 2024-04-13T13:35:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_6.export 2024-04-13T13:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.export 2024-04-13T13:35:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_6.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.export 2024-04-13T13:35:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_6.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_6.export 2024-04-13T13:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_7 2024-04-13T13:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_7 snap 2024-04-13T13:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:35:12.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2024-04-13T13:35:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-04-13T13:35:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-04-13T13:35:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:35:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-04-13T13:35:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:35:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:35:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:35:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-04-13T13:35:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:35:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:35:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:35:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:35:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:35:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:35:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:35:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:35:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:35:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:35:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-04-13T13:35:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-13T13:35:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:35:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:35:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-04-13T13:35:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:35:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:35:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:35:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:35:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-04-13T13:35:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-13T13:35:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:35:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:35:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:35:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:35:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:35:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:35:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:35:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:35:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:35:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-04-13T13:35:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:35:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:35:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:35:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:35:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:35:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_7.status 2024-04-13T13:35:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_7.status 2024-04-13T13:35:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_7.status 2024-04-13T13:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_7.status 2024-04-13T13:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_7.status 2024-04-13T13:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.status 2024-04-13T13:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-04-13T13:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-04-13T13:35:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 131 2024-04-13T13:35:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_7.status 2024-04-13T13:35:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_7.status 2024-04-13T13:35:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-04-13T13:35:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:35:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=131 2024-04-13T13:35:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:35:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.status 2024-04-13T13:35:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.status 2024-04-13T13:35:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-13T13:35:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 131 = '' 2024-04-13T13:35:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.status 2024-04-13T13:35:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.status 2024-04-13T13:35:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-13T13:35:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 131 = '' 2024-04-13T13:35:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.status 2024-04-13T13:35:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.status 2024-04-13T13:35:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-13T13:35:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=131 2024-04-13T13:35:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 131 = 131 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:18.017 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2024-04-13T13:35:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-04-13T13:35:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_7': 2024-04-13T13:35:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:35:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c1410de1b 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c1410de1b 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:21 2024 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:21 2024 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:21 2024 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 5e948815-4bc4-4300-914c-1855c84e9f33 2024-04-13T13:35:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:35:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_7 snap 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_7 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_7.export 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.export 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_7.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.export 2024-04-13T13:35:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_7.export 2024-04-13T13:35:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.export 2024-04-13T13:35:18.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8162479112624362) > weight (0.25) 2024-04-13T13:35:18.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8884085875540796) > weight (0.25) 2024-04-13T13:35:18.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6047484000199382) > weight (0.25) 2024-04-13T13:35:18.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6433867441501747) > weight (0.25) 2024-04-13T13:35:18.731 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-04-13T13:35:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_7.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.export 2024-04-13T13:35:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_7.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_7.export 2024-04-13T13:35:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:35:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_8 2024-04-13T13:35:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_8 snap 2024-04-13T13:35:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:35:20.749 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:35:20.757+0000 7f59186a3640 -1 reset not still connected to 0x55ea1803e340 2024-04-13T13:35:20.749 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:35:20.757+0000 7f59186a3640 -1 reset not still connected to 0x55ea18ed09c0 2024-04-13T13:35:20.749 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:35:20.757+0000 7f59186a3640 -1 reset not still connected to 0x55ea18ed12b0 2024-04-13T13:35:20.749 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:35:20.757+0000 7f59186a3640 -1 reset not still connected to 0x55ea18fafa00 2024-04-13T13:35:20.750 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:35:20.758+0000 7f59186a3640 -1 reset not still connected to 0x55ea20bac410 2024-04-13T13:35:20.751 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:35:20.759+0000 7f59186a3640 -1 reset not still connected to 0x55ea39d1add0 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-04-13T13:35:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:35:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:35:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:35:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:35:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:35:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-04-13T13:35:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:35:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:35:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:35:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-04-13T13:35:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:35:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:35:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:35:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:35:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-04-13T13:35:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_8.status 2024-04-13T13:35:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_8.status 2024-04-13T13:35:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_8.status 2024-04-13T13:35:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_8.status 2024-04-13T13:35:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_8.status 2024-04-13T13:35:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.status 2024-04-13T13:35:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-04-13T13:35:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-04-13T13:35:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 133 2024-04-13T13:35:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_8.status 2024-04-13T13:35:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_8.status 2024-04-13T13:35:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-04-13T13:35:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:35:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=133 2024-04-13T13:35:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:35:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.status 2024-04-13T13:35:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.status 2024-04-13T13:35:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-13T13:35:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 133 = '' 2024-04-13T13:35:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:23.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2024-04-13T13:35:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.status 2024-04-13T13:35:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.status 2024-04-13T13:35:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-13T13:35:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 133 = '' 2024-04-13T13:35:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.status 2024-04-13T13:35:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.status 2024-04-13T13:35:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-13T13:35:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=133 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 133 = 133 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-04-13T13:35:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_8': 2024-04-13T13:35:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:35:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:35:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:35:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b382b7609e 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b382b7609e 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:34 2024 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:34 2024 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:34 2024 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 970b3a1e-6448-44f1-be2c-e3b2bc2b08af 2024-04-13T13:35:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_8 snap 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_8 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_8.export 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.export 2024-04-13T13:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_8.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.export 2024-04-13T13:35:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_8.export 2024-04-13T13:35:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.export 2024-04-13T13:35:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_8.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.export 2024-04-13T13:35:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_8.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_8.export 2024-04-13T13:35:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:35:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_9 2024-04-13T13:35:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_9 snap 2024-04-13T13:35:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:35:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-04-13T13:35:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-04-13T13:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-04-13T13:35:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:35:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:35:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:35:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:35:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:35:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-04-13T13:35:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:35:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:35:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:35:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-04-13T13:35:28.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2024-04-13T13:35:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-13T13:35:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:35:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:35:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-04-13T13:35:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:35:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:35:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:35:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:35:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:35:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:35:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_9.status 2024-04-13T13:35:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_9.status 2024-04-13T13:35:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_9.status 2024-04-13T13:35:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_9.status 2024-04-13T13:35:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_9.status 2024-04-13T13:35:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.status 2024-04-13T13:35:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-04-13T13:35:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-04-13T13:35:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 135 2024-04-13T13:35:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_9.status 2024-04-13T13:35:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_9.status 2024-04-13T13:35:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-04-13T13:35:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:35:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=135 2024-04-13T13:35:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:35:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.status 2024-04-13T13:35:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.status 2024-04-13T13:35:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-13T13:35:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 135 = '' 2024-04-13T13:35:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.status 2024-04-13T13:35:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.status 2024-04-13T13:35:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-13T13:35:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 135 = '' 2024-04-13T13:35:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.status 2024-04-13T13:35:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.status 2024-04-13T13:35:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-13T13:35:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=135 2024-04-13T13:35:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 135 = 135 2024-04-13T13:35:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-04-13T13:35:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-04-13T13:35:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_9': 2024-04-13T13:35:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:35:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:35:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab8dfa0264 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab8dfa0264 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:41 2024 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:41 2024 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:41 2024 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 9b0f1c92-41a8-4559-8a23-3818c789553a 2024-04-13T13:35:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:35:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_9 snap 2024-04-13T13:35:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_9 2024-04-13T13:35:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:35:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_9.export 2024-04-13T13:35:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.export 2024-04-13T13:35:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_9.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.export 2024-04-13T13:35:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_9.export 2024-04-13T13:35:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.export 2024-04-13T13:35:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_9.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.export 2024-04-13T13:35:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_9.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_9.export 2024-04-13T13:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_10 2024-04-13T13:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_10 snap 2024-04-13T13:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:35:34.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-04-13T13:35:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:35:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:35:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:35:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-04-13T13:35:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:35:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:35:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:35:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:35:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:35:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:35:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-04-13T13:35:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-13T13:35:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:35:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:35:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:35:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-04-13T13:35:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:35:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:35:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:35:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:35:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-04-13T13:35:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_10.status 2024-04-13T13:35:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_10.status 2024-04-13T13:35:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_10.status 2024-04-13T13:35:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_10.status 2024-04-13T13:35:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_10.status 2024-04-13T13:35:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.status 2024-04-13T13:35:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-04-13T13:35:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-04-13T13:35:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 137 2024-04-13T13:35:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_10.status 2024-04-13T13:35:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_10.status 2024-04-13T13:35:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-04-13T13:35:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:35:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=137 2024-04-13T13:35:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:35:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.status 2024-04-13T13:35:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.status 2024-04-13T13:35:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-13T13:35:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 137 = '' 2024-04-13T13:35:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.status 2024-04-13T13:35:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.status 2024-04-13T13:35:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-13T13:35:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 137 = '' 2024-04-13T13:35:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.status 2024-04-13T13:35:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.status 2024-04-13T13:35:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-13T13:35:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=137 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 137 = 137 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_10': 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0e34d4a0e 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0e34d4a0e 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:35:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:56 2024 2024-04-13T13:35:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:26:56 2024 2024-04-13T13:35:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:56 2024 2024-04-13T13:35:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:35:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:35:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:35:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 7cb77a9f-0f4a-4ba9-bc4b-ae0678719819 2024-04-13T13:35:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:35:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_10 snap 2024-04-13T13:35:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_10 2024-04-13T13:35:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:35:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_10.export 2024-04-13T13:35:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.export 2024-04-13T13:35:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_10.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.export 2024-04-13T13:35:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_10.export 2024-04-13T13:35:39.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2024-04-13T13:35:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.export 2024-04-13T13:35:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_10.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.export 2024-04-13T13:35:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_10.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_10.export 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_11 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_11 snap 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:35:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-04-13T13:35:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:35:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:35:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:35:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-04-13T13:35:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:35:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:35:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:35:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:35:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:35:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:35:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:35:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:35:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-04-13T13:35:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:35:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:35:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:35:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:35:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:35:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-04-13T13:35:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-13T13:35:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:35:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:35:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-04-13T13:35:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:35:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:35:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:35:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-04-13T13:35:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:35:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:35:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:35:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:35:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:35:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:35:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_11.status 2024-04-13T13:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_11.status 2024-04-13T13:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_11.status 2024-04-13T13:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_11.status 2024-04-13T13:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_11.status 2024-04-13T13:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.status 2024-04-13T13:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-04-13T13:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-04-13T13:35:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 139 2024-04-13T13:35:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_11.status 2024-04-13T13:35:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_11.status 2024-04-13T13:35:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-04-13T13:35:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:35:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=139 2024-04-13T13:35:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:35:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.status 2024-04-13T13:35:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.status 2024-04-13T13:35:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-13T13:35:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 139 = '' 2024-04-13T13:35:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.status 2024-04-13T13:35:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.status 2024-04-13T13:35:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-13T13:35:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 139 = '' 2024-04-13T13:35:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:45.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2024-04-13T13:35:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.status 2024-04-13T13:35:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.status 2024-04-13T13:35:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-13T13:35:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=139 2024-04-13T13:35:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 139 = 139 2024-04-13T13:35:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-04-13T13:35:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_11': 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c21fe77c7 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c21fe77c7 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:35:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:04 2024 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:04 2024 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:04 2024 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 59ed4044-a6c7-40ba-a8ff-0113c71c89da 2024-04-13T13:35:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:35:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_11 snap 2024-04-13T13:35:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_11 2024-04-13T13:35:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:35:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_11.export 2024-04-13T13:35:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.export 2024-04-13T13:35:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_11.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.export 2024-04-13T13:35:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_11.export 2024-04-13T13:35:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.export 2024-04-13T13:35:46.836 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7440914255889869) > weight (0.25) 2024-04-13T13:35:46.836 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-13T13:35:46.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-13T13:35:46.837 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-13T13:35:46.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:2024-04-13T13:35:46.846+0000 7fc4441f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 53392) UID: 1000 2024-04-13T13:35:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_11.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.export 2024-04-13T13:35:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_11.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_11.export 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_12 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_12 snap 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:35:47.838 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-13T13:35:47.838 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:35:47.838 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-13T13:35:47.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-13T13:35:47.839 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-13T13:35:47.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-13T13:35:47.842 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-04-13T13:35:47.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi122.stderr:2024-04-13T13:35:47.956+0000 7fcf533f8640 -1 rbd::mirror::ImageReplayer: 0x56304702ec80 [3/e58af553-340d-46ba-a87f-856f475da231] register_admin_socket_hook: error registering admin socket commands 2024-04-13T13:35:47.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi122.stderr:2024-04-13T13:35:47.956+0000 7fcf52bf7640 -1 rbd::mirror::ImageReplayer: 0x56304702ec80 [3/e58af553-340d-46ba-a87f-856f475da231] register_admin_socket_hook: error registering admin socket commands 2024-04-13T13:35:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-04-13T13:35:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-04-13T13:35:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:35:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-04-13T13:35:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:35:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:35:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:35:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:35:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:35:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-04-13T13:35:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-13T13:35:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:35:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:35:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-04-13T13:35:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:35:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:35:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:35:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:35:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:35:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-04-13T13:35:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-13T13:35:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:35:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:35:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:35:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:35:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:35:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:35:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:35:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:35:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:35:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-04-13T13:35:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:35:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:35:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:35:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:35:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-04-13T13:35:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_12.status 2024-04-13T13:35:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_12.status 2024-04-13T13:35:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_12.status 2024-04-13T13:35:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_12.status 2024-04-13T13:35:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_12.status 2024-04-13T13:35:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.status 2024-04-13T13:35:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-04-13T13:35:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-04-13T13:35:50.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2024-04-13T13:35:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 141 2024-04-13T13:35:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_12.status 2024-04-13T13:35:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_12.status 2024-04-13T13:35:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-04-13T13:35:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:35:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=141 2024-04-13T13:35:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.status 2024-04-13T13:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.status 2024-04-13T13:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-13T13:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 141 = '' 2024-04-13T13:35:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.status 2024-04-13T13:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.status 2024-04-13T13:35:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-13T13:35:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 141 = '' 2024-04-13T13:35:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.status 2024-04-13T13:35:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.status 2024-04-13T13:35:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-13T13:35:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=141 2024-04-13T13:35:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 141 = 141 2024-04-13T13:35:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-04-13T13:35:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-04-13T13:35:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_12': 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b3bdf2e083 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b3bdf2e083 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:10 2024 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:10 2024 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:10 2024 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:35:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:35:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:35:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 5b84301a-f350-4908-84da-ddb3ac371259 2024-04-13T13:35:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:35:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_12 snap 2024-04-13T13:35:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_12 2024-04-13T13:35:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:35:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_12.export 2024-04-13T13:35:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.export 2024-04-13T13:35:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_12.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.export 2024-04-13T13:35:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_12.export 2024-04-13T13:35:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.export 2024-04-13T13:35:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:35:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_12.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.export 2024-04-13T13:35:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_12.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_12.export 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_13 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_13 snap 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-04-13T13:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-04-13T13:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:35:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:35:55.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2024-04-13T13:35:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-04-13T13:35:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:35:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:35:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:35:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-04-13T13:35:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:35:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:35:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:35:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:35:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:35:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:35:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:35:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:35:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:35:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:35:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-04-13T13:35:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:35:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:35:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:35:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:35:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:35:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:35:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:35:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:35:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:35:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-04-13T13:35:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:35:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:35:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:35:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:35:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:35:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:35:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:35:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-04-13T13:35:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_13.status 2024-04-13T13:35:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_13.status 2024-04-13T13:35:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_13.status 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_13.status 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_13.status 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.status 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-04-13T13:35:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 143 2024-04-13T13:35:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_13.status 2024-04-13T13:35:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:35:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_13.status 2024-04-13T13:35:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-04-13T13:35:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:35:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=143 2024-04-13T13:35:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:35:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.status 2024-04-13T13:35:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.status 2024-04-13T13:35:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-13T13:35:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 143 = '' 2024-04-13T13:35:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:35:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.status 2024-04-13T13:35:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.status 2024-04-13T13:35:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-13T13:35:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:35:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:35:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:35:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:35:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:35:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 143 = '' 2024-04-13T13:35:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:35:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:35:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.status 2024-04-13T13:35:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.status 2024-04-13T13:35:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-13T13:35:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:35:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:35:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=143 2024-04-13T13:35:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 143 = 143 2024-04-13T13:35:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:35:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-04-13T13:35:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:35:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:35:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:35:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:35:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:35:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:35:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-04-13T13:36:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_13': 2024-04-13T13:36:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:36:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:36:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:36:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10abb6a353e2 2024-04-13T13:36:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10abb6a353e2 2024-04-13T13:36:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:18 2024 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:18 2024 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:18 2024 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: bf426828-2e69-4ba4-a59f-e3526fdc760d 2024-04-13T13:36:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_13 snap 2024-04-13T13:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_13 2024-04-13T13:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_13.export 2024-04-13T13:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.export 2024-04-13T13:36:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_13.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.export 2024-04-13T13:36:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_13.export 2024-04-13T13:36:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.export 2024-04-13T13:36:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_13.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.export 2024-04-13T13:36:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_13.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_13.export 2024-04-13T13:36:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:36:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_14 2024-04-13T13:36:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_14 snap 2024-04-13T13:36:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:36:01.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2024-04-13T13:36:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-04-13T13:36:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:36:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:36:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:36:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:36:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-04-13T13:36:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-13T13:36:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-04-13T13:36:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:36:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:36:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:36:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:36:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-04-13T13:36:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-13T13:36:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-04-13T13:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:36:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-04-13T13:36:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:36:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:36:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:36:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:36:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:36:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_14.status 2024-04-13T13:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_14.status 2024-04-13T13:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_14.status 2024-04-13T13:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_14.status 2024-04-13T13:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_14.status 2024-04-13T13:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.status 2024-04-13T13:36:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-04-13T13:36:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-04-13T13:36:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 145 2024-04-13T13:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_14.status 2024-04-13T13:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_14.status 2024-04-13T13:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-04-13T13:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:36:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=145 2024-04-13T13:36:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:36:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.status 2024-04-13T13:36:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.status 2024-04-13T13:36:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-13T13:36:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 145 = '' 2024-04-13T13:36:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:36:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.status 2024-04-13T13:36:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.status 2024-04-13T13:36:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-13T13:36:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 145 = '' 2024-04-13T13:36:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:36:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.status 2024-04-13T13:36:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.status 2024-04-13T13:36:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-13T13:36:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=145 2024-04-13T13:36:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 145 = 145 2024-04-13T13:36:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-04-13T13:36:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:36:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:06.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2024-04-13T13:36:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-04-13T13:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_14': 2024-04-13T13:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0ecc4435e 2024-04-13T13:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0ecc4435e 2024-04-13T13:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:30 2024 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:30 2024 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:30 2024 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: dc2077e0-ec0a-4427-ab08-fc9e2fa06c71 2024-04-13T13:36:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:36:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_14 snap 2024-04-13T13:36:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_14 2024-04-13T13:36:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:36:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_14.export 2024-04-13T13:36:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.export 2024-04-13T13:36:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_14.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.export 2024-04-13T13:36:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_14.export 2024-04-13T13:36:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.export 2024-04-13T13:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_14.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.export 2024-04-13T13:36:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_14.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_14.export 2024-04-13T13:36:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:36:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_15 2024-04-13T13:36:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_15 snap 2024-04-13T13:36:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:36:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-04-13T13:36:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-04-13T13:36:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:36:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:36:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:36:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-04-13T13:36:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:36:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:36:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:36:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:36:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:36:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:36:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-04-13T13:36:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:36:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:36:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:36:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:36:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:36:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:36:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_15.status 2024-04-13T13:36:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_15.status 2024-04-13T13:36:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_15.status 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_15.status 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_15.status 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.status 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-04-13T13:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 147 2024-04-13T13:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_15.status 2024-04-13T13:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_15.status 2024-04-13T13:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-04-13T13:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:36:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=147 2024-04-13T13:36:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:36:12.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2024-04-13T13:36:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.status 2024-04-13T13:36:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.status 2024-04-13T13:36:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-13T13:36:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 147 = '' 2024-04-13T13:36:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:36:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.status 2024-04-13T13:36:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.status 2024-04-13T13:36:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-13T13:36:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 147 = '' 2024-04-13T13:36:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:36:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.status 2024-04-13T13:36:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.status 2024-04-13T13:36:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-13T13:36:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=147 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 147 = 147 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:36:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-04-13T13:36:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_15': 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124cdadfabd0 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124cdadfabd0 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:42 2024 2024-04-13T13:36:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:42 2024 2024-04-13T13:36:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:42 2024 2024-04-13T13:36:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:36:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:36:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:36:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: f38ae892-421e-4484-b8f6-de0cb55a7833 2024-04-13T13:36:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:36:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_15 snap 2024-04-13T13:36:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_15 2024-04-13T13:36:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:36:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_15.export 2024-04-13T13:36:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.export 2024-04-13T13:36:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_15.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.export 2024-04-13T13:36:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_15.export 2024-04-13T13:36:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.export 2024-04-13T13:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_15.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.export 2024-04-13T13:36:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_15.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_15.export 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_16 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_16 snap 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:36:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:17.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2024-04-13T13:36:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-04-13T13:36:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:36:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:36:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:36:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-04-13T13:36:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-04-13T13:36:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-04-13T13:36:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:36:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:36:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:36:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-04-13T13:36:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:36:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:36:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:36:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-04-13T13:36:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-04-13T13:36:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:36:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:36:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-04-13T13:36:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 1 2024-04-13T13:36:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-04-13T13:36:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-04-13T13:36:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-04-13T13:36:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:36:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:36:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:36:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:36:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-04-13T13:36:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:36:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:36:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:36:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-04-13T13:36:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:36:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:36:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:36:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-04-13T13:36:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 2 2024-04-13T13:36:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-04-13T13:36:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-04-13T13:36:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-04-13T13:36:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:36:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:36:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:36:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:36:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-04-13T13:36:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:36:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:36:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:36:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-04-13T13:36:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-13T13:36:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:36:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:36:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:36:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:36:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:36:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:36:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:36:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:36:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:36:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:36:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:36:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:36:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:36:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-04-13T13:36:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:36:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_16.status 2024-04-13T13:36:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_16.status 2024-04-13T13:36:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_16.status 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_16.status 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_16.status 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.status 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-04-13T13:36:22.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2024-04-13T13:36:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 149 2024-04-13T13:36:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_16.status 2024-04-13T13:36:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_16.status 2024-04-13T13:36:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-04-13T13:36:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:36:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=149 2024-04-13T13:36:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:36:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.status 2024-04-13T13:36:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.status 2024-04-13T13:36:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-13T13:36:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 149 = '' 2024-04-13T13:36:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:36:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.status 2024-04-13T13:36:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.status 2024-04-13T13:36:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-13T13:36:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 149 = '' 2024-04-13T13:36:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:36:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.status 2024-04-13T13:36:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.status 2024-04-13T13:36:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-13T13:36:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=149 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 149 = 149 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:36:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-04-13T13:36:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_16': 2024-04-13T13:36:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:36:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:36:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b3f0d72789 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b3f0d72789 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:49 2024 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:49 2024 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:49 2024 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 53de06e5-2f50-4532-ba7a-3aa9dd41bcfd 2024-04-13T13:36:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_16 snap 2024-04-13T13:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_16 2024-04-13T13:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_16.export 2024-04-13T13:36:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.export 2024-04-13T13:36:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_16.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.export 2024-04-13T13:36:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_16.export 2024-04-13T13:36:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.export 2024-04-13T13:36:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_16.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.export 2024-04-13T13:36:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_16.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_16.export 2024-04-13T13:36:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:36:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_17 2024-04-13T13:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_17 snap 2024-04-13T13:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:36:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-04-13T13:36:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-04-13T13:36:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:36:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:28.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2024-04-13T13:36:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-04-13T13:36:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:36:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:36:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-04-13T13:36:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-13T13:36:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:36:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:36:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:36:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-04-13T13:36:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-13T13:36:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:36:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:36:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:36:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-04-13T13:36:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-13T13:36:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:36:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:36:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:36:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:36:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:36:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2024-04-13T13:36:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:36:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:36:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:36:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:36:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:36:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:36:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:36:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-04-13T13:36:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_17.status 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_17.status 2024-04-13T13:36:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_17.status 2024-04-13T13:36:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_17.status 2024-04-13T13:36:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_17.status 2024-04-13T13:36:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-04-13T13:36:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-04-13T13:36:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 151 2024-04-13T13:36:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_17.status 2024-04-13T13:36:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_17.status 2024-04-13T13:36:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-04-13T13:36:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:36:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=151 2024-04-13T13:36:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:36:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-13T13:36:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=84 2024-04-13T13:36:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 151 = 84 2024-04-13T13:36:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:36:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-13T13:36:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=84 2024-04-13T13:36:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 151 = 84 2024-04-13T13:36:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:36:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-13T13:36:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 151 = '' 2024-04-13T13:36:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.status 2024-04-13T13:36:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-13T13:36:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=151 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 151 = 151 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:36:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:33.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2024-04-13T13:36:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_17': 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab656b9041 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab656b9041 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:36:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:59 2024 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:27:59 2024 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:59 2024 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: d2b64f41-9de5-4eab-8f44-349912e93682 2024-04-13T13:36:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_17 snap 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_17 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_17.export 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.export 2024-04-13T13:36:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_17.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.export 2024-04-13T13:36:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_17.export 2024-04-13T13:36:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.export 2024-04-13T13:36:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_17.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.export 2024-04-13T13:36:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_17.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_17.export 2024-04-13T13:36:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:36:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_18 2024-04-13T13:36:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_18 snap 2024-04-13T13:36:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:36:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-04-13T13:36:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-04-13T13:36:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:36:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-04-13T13:36:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:36:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:36:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:36:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-04-13T13:36:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-13T13:36:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-04-13T13:36:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:36:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:36:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:36:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-04-13T13:36:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-13T13:36:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:36:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:36:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:36:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:36:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-04-13T13:36:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:36:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:36:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:36:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:36:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_18.status 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_18.status 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_18.status 2024-04-13T13:36:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_18.status 2024-04-13T13:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_18.status 2024-04-13T13:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.status 2024-04-13T13:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-04-13T13:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-04-13T13:36:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 153 2024-04-13T13:36:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_18.status 2024-04-13T13:36:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_18.status 2024-04-13T13:36:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-04-13T13:36:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:36:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=153 2024-04-13T13:36:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:36:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.status 2024-04-13T13:36:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.status 2024-04-13T13:36:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-13T13:36:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=85 2024-04-13T13:36:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 153 = 85 2024-04-13T13:36:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:36:39.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2024-04-13T13:36:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.status 2024-04-13T13:36:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.status 2024-04-13T13:36:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-13T13:36:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 153 = '' 2024-04-13T13:36:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:36:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.status 2024-04-13T13:36:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.status 2024-04-13T13:36:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-13T13:36:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=153 2024-04-13T13:36:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 153 = 153 2024-04-13T13:36:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-04-13T13:36:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:36:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_18': 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0daec96fa 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0daec96fa 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:11 2024 2024-04-13T13:36:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:11 2024 2024-04-13T13:36:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:11 2024 2024-04-13T13:36:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:36:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:36:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:36:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 807d8287-a59d-4634-8f59-0370463ec872 2024-04-13T13:36:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:36:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_18 snap 2024-04-13T13:36:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_18 2024-04-13T13:36:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:36:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_18.export 2024-04-13T13:36:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.export 2024-04-13T13:36:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_18.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.export 2024-04-13T13:36:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_18.export 2024-04-13T13:36:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.export 2024-04-13T13:36:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_18.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.export 2024-04-13T13:36:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_18.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_18.export 2024-04-13T13:36:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:36:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_19 2024-04-13T13:36:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_19 snap 2024-04-13T13:36:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-04-13T13:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-04-13T13:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:44.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2024-04-13T13:36:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-04-13T13:36:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:36:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:36:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:36:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-04-13T13:36:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:36:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:36:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:36:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:36:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:36:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:36:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-04-13T13:36:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-13T13:36:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-04-13T13:36:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:36:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:36:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:36:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:36:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:36:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:36:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-04-13T13:36:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:36:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:36:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:36:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:36:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:36:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_19.status 2024-04-13T13:36:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_19.status 2024-04-13T13:36:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_19.status 2024-04-13T13:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_19.status 2024-04-13T13:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_19.status 2024-04-13T13:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-04-13T13:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-04-13T13:36:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 155 2024-04-13T13:36:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_19.status 2024-04-13T13:36:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_19.status 2024-04-13T13:36:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-04-13T13:36:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=155 2024-04-13T13:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:36:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-13T13:36:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 155 = '' 2024-04-13T13:36:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:36:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-13T13:36:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 155 = '' 2024-04-13T13:36:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:36:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-13T13:36:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 155 = '' 2024-04-13T13:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:36:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.status 2024-04-13T13:36:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-13T13:36:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=155 2024-04-13T13:36:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 155 = 155 2024-04-13T13:36:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-04-13T13:36:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:36:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:49.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2024-04-13T13:36:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-04-13T13:36:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_19': 2024-04-13T13:36:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:36:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:36:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124ced325f9d 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124ced325f9d 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:18 2024 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:18 2024 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:18 2024 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 0c18f79e-f883-47a8-84ef-a7789e73b4ba 2024-04-13T13:36:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:36:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_19 snap 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_19 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_19.export 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.export 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_19.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.export 2024-04-13T13:36:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_19.export 2024-04-13T13:36:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.export 2024-04-13T13:36:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_19.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.export 2024-04-13T13:36:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_19.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_19.export 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_20 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_20 snap 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:36:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-04-13T13:36:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:36:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:36:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-04-13T13:36:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-04-13T13:36:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:36:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:36:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:36:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:36:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:36:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:36:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:36:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_20.status 2024-04-13T13:36:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_20.status 2024-04-13T13:36:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_20.status 2024-04-13T13:36:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_20.status 2024-04-13T13:36:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_20.status 2024-04-13T13:36:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:36:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.status 2024-04-13T13:36:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-04-13T13:36:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-04-13T13:36:53.731 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-13T13:36:53.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-13T13:36:53.732 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-13T13:36:53.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr:2024-04-13T13:36:53.741+0000 7f698f9f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 51874) UID: 1000 2024-04-13T13:36:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 157 2024-04-13T13:36:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_20.status 2024-04-13T13:36:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_20.status 2024-04-13T13:36:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-04-13T13:36:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:36:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=157 2024-04-13T13:36:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:36:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.status 2024-04-13T13:36:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.status 2024-04-13T13:36:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-13T13:36:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 157 = '' 2024-04-13T13:36:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:36:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.status 2024-04-13T13:36:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.status 2024-04-13T13:36:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-13T13:36:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:36:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:36:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:36:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:36:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:36:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 157 = '' 2024-04-13T13:36:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:36:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:36:55.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2024-04-13T13:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.status 2024-04-13T13:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.status 2024-04-13T13:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-13T13:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:36:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:36:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=157 2024-04-13T13:36:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 157 = 157 2024-04-13T13:36:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-04-13T13:36:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:36:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:55.843 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-13T13:36:55.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-13T13:36:55.843 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-13T13:36:55.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi122.stderr:2024-04-13T13:36:55.852+0000 7ff1a99f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 46641) UID: 1000 2024-04-13T13:36:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-04-13T13:36:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_20': 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b39eb8c6ed 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b39eb8c6ed 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:28 2024 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:28 2024 2024-04-13T13:36:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:28 2024 2024-04-13T13:36:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:36:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:36:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:36:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: c3ae73db-b27b-4d07-9b28-0323537a736a 2024-04-13T13:36:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:36:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_20 snap 2024-04-13T13:36:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_20 2024-04-13T13:36:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:36:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_20.export 2024-04-13T13:36:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.export 2024-04-13T13:36:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_20.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.export 2024-04-13T13:36:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_20.export 2024-04-13T13:36:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.export 2024-04-13T13:36:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:36:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_20.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.export 2024-04-13T13:36:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_20.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_20.export 2024-04-13T13:36:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:36:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:36:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_21 2024-04-13T13:36:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_21 snap 2024-04-13T13:36:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:36:58.732 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-13T13:36:58.733 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:36:58.733 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-13T13:36:58.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-13T13:36:58.733 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-13T13:36:58.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-13T13:36:58.737 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-04-13T13:36:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-04-13T13:36:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:36:58.844 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-13T13:36:58.845 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:36:58.845 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-13T13:36:58.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-13T13:36:58.845 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-13T13:36:58.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-13T13:36:58.848 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-04-13T13:36:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-04-13T13:36:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:36:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:36:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:36:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:36:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:36:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:36:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-04-13T13:36:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:36:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:36:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:36:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:36:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:36:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:36:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:36:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:36:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:36:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:36:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:36:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:36:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-04-13T13:36:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-13T13:36:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:36:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:36:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:36:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-04-13T13:36:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:36:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:36:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:36:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:36:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:36:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-04-13T13:37:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-13T13:37:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:37:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:37:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:37:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:37:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-04-13T13:37:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-04-13T13:37:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_21.status 2024-04-13T13:37:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_21.status 2024-04-13T13:37:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_21.status 2024-04-13T13:37:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_21.status 2024-04-13T13:37:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_21.status 2024-04-13T13:37:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.status 2024-04-13T13:37:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-04-13T13:37:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-04-13T13:37:00.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2024-04-13T13:37:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 159 2024-04-13T13:37:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_21.status 2024-04-13T13:37:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_21.status 2024-04-13T13:37:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-04-13T13:37:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=159 2024-04-13T13:37:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.status 2024-04-13T13:37:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.status 2024-04-13T13:37:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-13T13:37:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 159 = '' 2024-04-13T13:37:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.status 2024-04-13T13:37:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.status 2024-04-13T13:37:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-13T13:37:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 159 = '' 2024-04-13T13:37:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.status 2024-04-13T13:37:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.status 2024-04-13T13:37:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-13T13:37:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=159 2024-04-13T13:37:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 159 = 159 2024-04-13T13:37:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-04-13T13:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_21': 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10abeb6b2184 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10abeb6b2184 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:33 2024 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:33 2024 2024-04-13T13:37:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:33 2024 2024-04-13T13:37:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:37:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:37:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:37:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 213e24cb-89ef-475f-84d5-92800a3d1efb 2024-04-13T13:37:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:37:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_21 snap 2024-04-13T13:37:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_21 2024-04-13T13:37:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:37:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_21.export 2024-04-13T13:37:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.export 2024-04-13T13:37:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_21.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.export 2024-04-13T13:37:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_21.export 2024-04-13T13:37:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.export 2024-04-13T13:37:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_21.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.export 2024-04-13T13:37:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_21.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_21.export 2024-04-13T13:37:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:37:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_22 2024-04-13T13:37:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_22 snap 2024-04-13T13:37:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:37:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:06.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2024-04-13T13:37:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-04-13T13:37:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:37:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:37:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:37:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-04-13T13:37:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:37:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:37:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:37:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:37:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:37:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:37:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-04-13T13:37:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:37:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:37:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:37:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:37:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:37:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-04-13T13:37:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-13T13:37:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-04-13T13:37:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:37:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:37:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:37:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:37:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-04-13T13:37:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-04-13T13:37:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_22.status 2024-04-13T13:37:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_22.status 2024-04-13T13:37:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_22.status 2024-04-13T13:37:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_22.status 2024-04-13T13:37:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_22.status 2024-04-13T13:37:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.status 2024-04-13T13:37:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-04-13T13:37:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-04-13T13:37:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 161 2024-04-13T13:37:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_22.status 2024-04-13T13:37:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_22.status 2024-04-13T13:37:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-04-13T13:37:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=161 2024-04-13T13:37:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.status 2024-04-13T13:37:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.status 2024-04-13T13:37:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-13T13:37:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 161 = '' 2024-04-13T13:37:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.status 2024-04-13T13:37:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.status 2024-04-13T13:37:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-13T13:37:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 161 = '' 2024-04-13T13:37:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.status 2024-04-13T13:37:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.status 2024-04-13T13:37:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-13T13:37:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=161 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 161 = 161 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:37:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-04-13T13:37:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_22': 2024-04-13T13:37:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e03f1c9867 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e03f1c9867 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:45 2024 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:45 2024 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:45 2024 2024-04-13T13:37:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:37:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:37:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:37:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: e58af553-340d-46ba-a87f-856f475da231 2024-04-13T13:37:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:37:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_22 snap 2024-04-13T13:37:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_22 2024-04-13T13:37:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:37:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_22.export 2024-04-13T13:37:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.export 2024-04-13T13:37:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_22.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.export 2024-04-13T13:37:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_22.export 2024-04-13T13:37:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.export 2024-04-13T13:37:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_22.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.export 2024-04-13T13:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_22.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_22.export 2024-04-13T13:37:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:37:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_23 2024-04-13T13:37:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_23 snap 2024-04-13T13:37:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:37:11.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2024-04-13T13:37:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-04-13T13:37:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-04-13T13:37:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:37:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:37:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:37:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-04-13T13:37:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:37:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:37:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:37:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:37:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:37:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:37:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:37:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:37:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-04-13T13:37:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:37:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:37:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:37:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:37:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-04-13T13:37:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-04-13T13:37:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:37:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:37:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:37:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-04-13T13:37:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_23.status 2024-04-13T13:37:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_23.status 2024-04-13T13:37:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_23.status 2024-04-13T13:37:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_23.status 2024-04-13T13:37:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_23.status 2024-04-13T13:37:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.status 2024-04-13T13:37:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-04-13T13:37:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-04-13T13:37:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 163 2024-04-13T13:37:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_23.status 2024-04-13T13:37:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_23.status 2024-04-13T13:37:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-04-13T13:37:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=163 2024-04-13T13:37:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.status 2024-04-13T13:37:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.status 2024-04-13T13:37:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-13T13:37:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 163 = '' 2024-04-13T13:37:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.status 2024-04-13T13:37:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.status 2024-04-13T13:37:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-13T13:37:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 163 = '' 2024-04-13T13:37:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.status 2024-04-13T13:37:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.status 2024-04-13T13:37:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-13T13:37:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=163 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 163 = 163 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:16.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2024-04-13T13:37:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_23': 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c1a2d37a 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c1a2d37a 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:37:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:52 2024 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:28:52 2024 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:52 2024 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8c509f7e-2bfc-4798-88c9-e8a036e7a227 2024-04-13T13:37:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_23 snap 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_23 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_23.export 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.export 2024-04-13T13:37:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_23.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.export 2024-04-13T13:37:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_23.export 2024-04-13T13:37:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.export 2024-04-13T13:37:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_23.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.export 2024-04-13T13:37:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_23.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_23.export 2024-04-13T13:37:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:37:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_24 2024-04-13T13:37:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_24 snap 2024-04-13T13:37:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-04-13T13:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-04-13T13:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:37:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:37:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:37:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:37:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:37:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:37:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-04-13T13:37:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-13T13:37:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:37:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:37:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:37:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-04-13T13:37:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-13T13:37:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:37:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:37:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:37:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:37:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-04-13T13:37:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-13T13:37:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2024-04-13T13:37:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:37:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:37:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:37:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:37:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:37:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:37:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:37:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:37:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:37:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:37:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:37:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:37:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:37:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:37:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2024-04-13T13:37:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_24.status 2024-04-13T13:37:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_24.status 2024-04-13T13:37:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_24.status 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_24.status 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_24.status 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.status 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-04-13T13:37:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 165 2024-04-13T13:37:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_24.status 2024-04-13T13:37:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_24.status 2024-04-13T13:37:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-04-13T13:37:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=165 2024-04-13T13:37:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.status 2024-04-13T13:37:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.status 2024-04-13T13:37:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-13T13:37:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 165 = '' 2024-04-13T13:37:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:22.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2024-04-13T13:37:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.status 2024-04-13T13:37:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.status 2024-04-13T13:37:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-13T13:37:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 165 = '' 2024-04-13T13:37:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.status 2024-04-13T13:37:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.status 2024-04-13T13:37:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-13T13:37:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=165 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 165 = 165 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:37:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_24': 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b3ce2403df 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b3ce2403df 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:37:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:01 2024 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:01 2024 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:01 2024 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 91c5558e-79d7-4410-a35b-b56664447d57 2024-04-13T13:37:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:37:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_24 snap 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_24 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_24.export 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.export 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_24.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.export 2024-04-13T13:37:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_24.export 2024-04-13T13:37:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.export 2024-04-13T13:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_24.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.export 2024-04-13T13:37:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_24.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_24.export 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_25 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_25 snap 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:37:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-04-13T13:37:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-04-13T13:37:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:37:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:27.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2024-04-13T13:37:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-04-13T13:37:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:37:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:37:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:37:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-04-13T13:37:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:37:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:37:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:37:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:37:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-04-13T13:37:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-13T13:37:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:37:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:37:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:37:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-04-13T13:37:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-13T13:37:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:37:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:37:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:37:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:37:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-04-13T13:37:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-04-13T13:37:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_25.status 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_25.status 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_25.status 2024-04-13T13:37:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_25.status 2024-04-13T13:37:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_25.status 2024-04-13T13:37:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.status 2024-04-13T13:37:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-04-13T13:37:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-04-13T13:37:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 167 2024-04-13T13:37:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_25.status 2024-04-13T13:37:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_25.status 2024-04-13T13:37:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-04-13T13:37:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=167 2024-04-13T13:37:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.status 2024-04-13T13:37:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.status 2024-04-13T13:37:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-13T13:37:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 167 = '' 2024-04-13T13:37:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.status 2024-04-13T13:37:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.status 2024-04-13T13:37:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-13T13:37:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:29.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 167 = '' 2024-04-13T13:37:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.status 2024-04-13T13:37:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.status 2024-04-13T13:37:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-13T13:37:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=167 2024-04-13T13:37:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 167 = 167 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-04-13T13:37:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_25': 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10abe9899b25 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10abe9899b25 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:07 2024 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:07 2024 2024-04-13T13:37:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:07 2024 2024-04-13T13:37:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:37:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:37:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:37:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 9c3874e8-22df-4fdb-a27a-7e2740eed944 2024-04-13T13:37:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:37:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_25 snap 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_25 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_25.export 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.export 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_25.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.export 2024-04-13T13:37:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_25.export 2024-04-13T13:37:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.export 2024-04-13T13:37:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_25.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.export 2024-04-13T13:37:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_25.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_25.export 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_26 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_26 snap 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:37:33.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2024-04-13T13:37:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-04-13T13:37:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:37:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:37:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:37:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:37:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:37:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:37:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:37:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:37:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-04-13T13:37:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-13T13:37:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-04-13T13:37:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:37:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:37:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:37:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-04-13T13:37:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-13T13:37:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:37:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:37:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:37:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-04-13T13:37:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_26.status 2024-04-13T13:37:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_26.status 2024-04-13T13:37:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_26.status 2024-04-13T13:37:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_26.status 2024-04-13T13:37:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_26.status 2024-04-13T13:37:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.status 2024-04-13T13:37:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-04-13T13:37:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-04-13T13:37:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 169 2024-04-13T13:37:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_26.status 2024-04-13T13:37:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_26.status 2024-04-13T13:37:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-04-13T13:37:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=169 2024-04-13T13:37:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.status 2024-04-13T13:37:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.status 2024-04-13T13:37:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-13T13:37:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 169 = '' 2024-04-13T13:37:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.status 2024-04-13T13:37:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.status 2024-04-13T13:37:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-13T13:37:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 169 = '' 2024-04-13T13:37:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:37.235 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:37:37.243+0000 7f8106e21640 -1 reset not still connected to 0x555d8ed92c30 2024-04-13T13:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.status 2024-04-13T13:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.status 2024-04-13T13:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-13T13:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=169 2024-04-13T13:37:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 169 = 169 2024-04-13T13:37:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-04-13T13:37:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:37:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:38.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2024-04-13T13:37:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-04-13T13:37:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_26': 2024-04-13T13:37:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:37:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0270e3823 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0270e3823 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:14 2024 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:14 2024 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:14 2024 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: cc11b282-5f39-432a-9e34-fd7d24025087 2024-04-13T13:37:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:37:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_26 snap 2024-04-13T13:37:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_26 2024-04-13T13:37:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:37:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_26.export 2024-04-13T13:37:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.export 2024-04-13T13:37:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_26.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.export 2024-04-13T13:37:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_26.export 2024-04-13T13:37:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.export 2024-04-13T13:37:38.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_26.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.export 2024-04-13T13:37:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_26.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_26.export 2024-04-13T13:37:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:37:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_27 2024-04-13T13:37:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_27 snap 2024-04-13T13:37:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:37:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-04-13T13:37:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:37:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:37:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:37:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:37:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:37:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:37:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:37:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:37:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:37:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-04-13T13:37:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-13T13:37:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-04-13T13:37:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:37:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:37:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:37:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:37:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-04-13T13:37:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-13T13:37:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:37:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-04-13T13:37:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-04-13T13:37:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_27.status 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_27.status 2024-04-13T13:37:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_27.status 2024-04-13T13:37:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_27.status 2024-04-13T13:37:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_27.status 2024-04-13T13:37:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.status 2024-04-13T13:37:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-04-13T13:37:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-04-13T13:37:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 171 2024-04-13T13:37:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_27.status 2024-04-13T13:37:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_27.status 2024-04-13T13:37:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-04-13T13:37:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=171 2024-04-13T13:37:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.status 2024-04-13T13:37:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.status 2024-04-13T13:37:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-13T13:37:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 171 = '' 2024-04-13T13:37:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.status 2024-04-13T13:37:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.status 2024-04-13T13:37:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-13T13:37:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 171 = '' 2024-04-13T13:37:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:43.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2024-04-13T13:37:44.130 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:37:44.138+0000 7f8106e21640 -1 reset not still connected to 0x555d47e87ad0 2024-04-13T13:37:44.130 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:37:44.138+0000 7f8106e21640 -1 reset not still connected to 0x555d48f335f0 2024-04-13T13:37:44.130 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:37:44.139+0000 7f8106e21640 -1 reset not still connected to 0x555d49ad32b0 2024-04-13T13:37:44.132 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:37:44.140+0000 7f8106e21640 -1 reset not still connected to 0x555d4b80e0d0 2024-04-13T13:37:44.132 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:37:44.140+0000 7f8106e21640 -1 reset not still connected to 0x555d4b80e1a0 2024-04-13T13:37:44.132 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:37:44.140+0000 7f8106e21640 -1 reset not still connected to 0x555d4bc58410 2024-04-13T13:37:44.132 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:37:44.140+0000 7f8106e21640 -1 reset not still connected to 0x555d8ed92c30 2024-04-13T13:37:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.status 2024-04-13T13:37:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.status 2024-04-13T13:37:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-13T13:37:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=171 2024-04-13T13:37:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 171 = 171 2024-04-13T13:37:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-04-13T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-04-13T13:37:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_27': 2024-04-13T13:37:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:37:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:37:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:37:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c4cccc1c3 2024-04-13T13:37:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c4cccc1c3 2024-04-13T13:37:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:37:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:20 2024 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:20 2024 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:20 2024 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 7a6330ae-57ed-417b-a82d-4d987787e23b 2024-04-13T13:37:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:37:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_27 snap 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_27 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_27.export 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.export 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_27.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.export 2024-04-13T13:37:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_27.export 2024-04-13T13:37:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.export 2024-04-13T13:37:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_27.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.export 2024-04-13T13:37:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_27.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_27.export 2024-04-13T13:37:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:37:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_28 2024-04-13T13:37:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_28 snap 2024-04-13T13:37:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:37:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-04-13T13:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:37:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-04-13T13:37:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:37:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-04-13T13:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:37:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:37:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:37:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:37:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:37:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-04-13T13:37:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-13T13:37:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-04-13T13:37:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:37:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:37:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:37:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:37:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-04-13T13:37:49.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2024-04-13T13:37:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-13T13:37:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-04-13T13:37:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:37:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:37:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:37:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:37:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:37:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-04-13T13:37:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-13T13:37:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2024-04-13T13:37:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T13:37:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:3 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:3 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T13:37:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-13T13:37:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=3 2024-04-13T13:37:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=3 2024-04-13T13:37:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-13T13:37:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:37:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:37:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-13T13:37:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2024-04-13T13:37:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_28.status 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_28.status 2024-04-13T13:37:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_28.status 2024-04-13T13:37:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_28.status 2024-04-13T13:37:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_28.status 2024-04-13T13:37:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-04-13T13:37:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-04-13T13:37:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 173 2024-04-13T13:37:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_28.status 2024-04-13T13:37:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_28.status 2024-04-13T13:37:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-04-13T13:37:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=173 2024-04-13T13:37:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-13T13:37:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=95 2024-04-13T13:37:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 173 = 95 2024-04-13T13:37:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-13T13:37:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=95 2024-04-13T13:37:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 173 = 95 2024-04-13T13:37:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-13T13:37:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 173 = '' 2024-04-13T13:37:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:37:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.status 2024-04-13T13:37:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-13T13:37:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=173 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 173 = 173 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:37:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-04-13T13:37:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_28': 2024-04-13T13:37:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:37:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b38bced57c 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b38bced57c 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:27 2024 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:27 2024 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:27 2024 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 6c4ef21c-2a7c-465b-b23d-96050d4d8338 2024-04-13T13:37:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:37:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_28 snap 2024-04-13T13:37:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_28 2024-04-13T13:37:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:37:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_28.export 2024-04-13T13:37:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.export 2024-04-13T13:37:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_28.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.export 2024-04-13T13:37:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_28.export 2024-04-13T13:37:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.export 2024-04-13T13:37:54.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2024-04-13T13:37:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:37:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_28.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.export 2024-04-13T13:37:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_28.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_28.export 2024-04-13T13:37:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:37:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_29 2024-04-13T13:37:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_29 snap 2024-04-13T13:37:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:37:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:37:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:37:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-04-13T13:37:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:37:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:37:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:37:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-04-13T13:37:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:37:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:37:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:37:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:37:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:37:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:37:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:37:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:37:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:37:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-13T13:37:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:37:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:37:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:37:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-13T13:37:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-13T13:37:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:37:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:37:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:37:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:37:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-13T13:37:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-13T13:37:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:37:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-04-13T13:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:37:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:37:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:37:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:37:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:37:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-04-13T13:37:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:37:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:37:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:37:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:37:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-04-13T13:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_29.status 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_29.status 2024-04-13T13:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_29.status 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_29.status 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_29.status 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-04-13T13:37:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 175 2024-04-13T13:37:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_29.status 2024-04-13T13:37:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:37:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_29.status 2024-04-13T13:37:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-04-13T13:37:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:37:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=175 2024-04-13T13:37:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:37:58.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:37:58.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:37:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-13T13:37:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=96 2024-04-13T13:37:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 175 = 96 2024-04-13T13:37:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:37:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:37:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:37:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-13T13:37:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=96 2024-04-13T13:37:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 175 = 96 2024-04-13T13:37:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:37:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:37:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:37:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:37:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:37:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:37:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-13T13:37:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:37:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:37:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:37:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:37:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:37:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:37:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:37:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 175 = '' 2024-04-13T13:37:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:37:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:38:00.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2024-04-13T13:38:01.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:38:01.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:01.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:38:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.status 2024-04-13T13:38:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-13T13:38:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=175 2024-04-13T13:38:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 175 = 175 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:38:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:38:02.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_29': 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10aba9bacfc5 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10aba9bacfc5 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:38:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:39 2024 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:39 2024 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:39 2024 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: d1a11424-da29-4237-abbf-b187574ee6bd 2024-04-13T13:38:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:38:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_29 snap 2024-04-13T13:38:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_29 2024-04-13T13:38:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:38:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_29.export 2024-04-13T13:38:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.export 2024-04-13T13:38:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_29.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.export 2024-04-13T13:38:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_29.export 2024-04-13T13:38:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:38:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.export 2024-04-13T13:38:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:38:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_29.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.export 2024-04-13T13:38:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_29.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_29.export 2024-04-13T13:38:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:38:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_30 2024-04-13T13:38:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_30 snap 2024-04-13T13:38:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-04-13T13:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-04-13T13:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:38:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-04-13T13:38:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:38:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:38:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:38:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-04-13T13:38:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:38:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:38:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-04-13T13:38:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:38:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:38:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:38:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:38:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:38:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:38:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:38:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:38:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:38:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:38:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:38:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:38:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-04-13T13:38:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-13T13:38:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:38:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:38:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:38:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:38:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:38:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:38:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:38:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-04-13T13:38:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:38:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:38:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:38:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:38:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:38:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:38:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-04-13T13:38:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:38:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_30.status 2024-04-13T13:38:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_30.status 2024-04-13T13:38:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:38:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_30.status 2024-04-13T13:38:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_30.status 2024-04-13T13:38:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_30.status 2024-04-13T13:38:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:38:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-04-13T13:38:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-04-13T13:38:05.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2024-04-13T13:38:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 177 2024-04-13T13:38:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_30.status 2024-04-13T13:38:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_30.status 2024-04-13T13:38:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-04-13T13:38:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:38:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=177 2024-04-13T13:38:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:38:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:38:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-13T13:38:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=97 2024-04-13T13:38:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 177 = 97 2024-04-13T13:38:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:38:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-13T13:38:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=97 2024-04-13T13:38:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 177 = 97 2024-04-13T13:38:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:38:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-13T13:38:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:38:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:38:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:38:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:38:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:38:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 177 = '' 2024-04-13T13:38:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2024-04-13T13:38:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.status 2024-04-13T13:38:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-13T13:38:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=177 2024-04-13T13:38:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 177 = 177 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:38:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_30': 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e051dfcca0 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e051dfcca0 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:51 2024 2024-04-13T13:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:29:51 2024 2024-04-13T13:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:51 2024 2024-04-13T13:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: df429b8a-ec65-47e3-b367-41f75bbe4da1 2024-04-13T13:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_30 snap 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_30 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_30.export 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.export 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_30.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.export 2024-04-13T13:38:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_30.export 2024-04-13T13:38:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:38:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.export 2024-04-13T13:38:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:38:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_30.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.export 2024-04-13T13:38:10.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2024-04-13T13:38:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_30.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_30.export 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_31 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_31 snap 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:38:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-04-13T13:38:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-04-13T13:38:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:38:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-04-13T13:38:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:38:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:38:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:38:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:38:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:38:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:38:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:38:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:38:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:38:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:38:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:38:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:38:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:38:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:38:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:38:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-04-13T13:38:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-13T13:38:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:38:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:38:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:38:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:38:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:38:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:38:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-04-13T13:38:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-13T13:38:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:38:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:38:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-04-13T13:38:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:38:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:38:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:38:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:38:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:38:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:38:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-04-13T13:38:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:38:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:38:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:38:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:38:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:38:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:38:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:38:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-04-13T13:38:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:38:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_31.status 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_31.status 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:38:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_31.status 2024-04-13T13:38:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_31.status 2024-04-13T13:38:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_31.status 2024-04-13T13:38:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:38:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.status 2024-04-13T13:38:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-04-13T13:38:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-04-13T13:38:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 179 2024-04-13T13:38:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_31.status 2024-04-13T13:38:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_31.status 2024-04-13T13:38:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-04-13T13:38:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:38:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=179 2024-04-13T13:38:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:38:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:38:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.status 2024-04-13T13:38:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.status 2024-04-13T13:38:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-13T13:38:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:38:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:38:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:38:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:38:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:38:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 179 = '' 2024-04-13T13:38:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:38:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.status 2024-04-13T13:38:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.status 2024-04-13T13:38:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-13T13:38:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:38:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:38:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:38:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:38:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:38:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 179 = '' 2024-04-13T13:38:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:38:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.status 2024-04-13T13:38:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.status 2024-04-13T13:38:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-13T13:38:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:16.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2024-04-13T13:38:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=179 2024-04-13T13:38:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 179 = 179 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:38:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-04-13T13:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_31': 2024-04-13T13:38:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:38:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:38:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:38:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c2ea13d0c7 2024-04-13T13:38:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c2ea13d0c7 2024-04-13T13:38:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:38:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:00 2024 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:00 2024 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:00 2024 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: f666e830-e8d8-4600-a27f-f8027ab46f85 2024-04-13T13:38:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:38:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_31 snap 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_31 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_31.export 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.export 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_31.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.export 2024-04-13T13:38:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_31.export 2024-04-13T13:38:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:38:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.export 2024-04-13T13:38:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:38:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_31.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.export 2024-04-13T13:38:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_31.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_31.export 2024-04-13T13:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_32 2024-04-13T13:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_32 snap 2024-04-13T13:38:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:38:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-04-13T13:38:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-04-13T13:38:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:38:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:38:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:38:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:38:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-04-13T13:38:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:38:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:38:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:38:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:38:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:38:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:38:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:38:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:38:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:38:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-04-13T13:38:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:38:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:38:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:38:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:38:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:38:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:38:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_32.status 2024-04-13T13:38:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_32.status 2024-04-13T13:38:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_32.status 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_32.status 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_32.status 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.status 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-04-13T13:38:21.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2024-04-13T13:38:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Snapshot ID: 181 2024-04-13T13:38:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_32.status 2024-04-13T13:38:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_32.status 2024-04-13T13:38:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-04-13T13:38:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-13T13:38:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snapshot_id=181 2024-04-13T13:38:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:38:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2024-04-13T13:38:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.status 2024-04-13T13:38:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.status 2024-04-13T13:38:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-13T13:38:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:38:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:38:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:38:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:38:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:38:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 181 = '' 2024-04-13T13:38:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2024-04-13T13:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.status 2024-04-13T13:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.status 2024-04-13T13:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-13T13:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2024-04-13T13:38:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:-:1.1: Document is empty 2024-04-13T13:38:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2024-04-13T13:38:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:^ 2024-04-13T13:38:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id= 2024-04-13T13:38:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 181 = '' 2024-04-13T13:38:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-13T13:38:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2024-04-13T13:38:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.status 2024-04-13T13:38:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.status 2024-04-13T13:38:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-13T13:38:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-13T13:38:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-13T13:38:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local primary_snapshot_id=181 2024-04-13T13:38:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 181 = 181 2024-04-13T13:38:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-04-13T13:38:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:38:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-13T13:38:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:38:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-04-13T13:38:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_32': 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c2cefedd3b 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c2cefedd3b 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:12 2024 2024-04-13T13:38:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:12 2024 2024-04-13T13:38:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:12 2024 2024-04-13T13:38:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2024-04-13T13:38:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:38:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:38:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: c95975b5-0eee-4f76-8be8-10a90b792c79 2024-04-13T13:38:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:38:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_snaps mirror image_32 snap 2024-04-13T13:38:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_32 2024-04-13T13:38:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2024-04-13T13:38:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_32.export 2024-04-13T13:38:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.export 2024-04-13T13:38:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_32.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.export 2024-04-13T13:38:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_32.export 2024-04-13T13:38:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:38:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.export 2024-04-13T13:38:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-13T13:38:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_32.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.export 2024-04-13T13:38:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.6pyizPAJcx/cluster2-mirror-image_32.export /tmp/tmp.6pyizPAJcx/cluster1-mirror-image_32.export 2024-04-13T13:38:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-13T13:38:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=image_33 2024-04-13T13:38:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snap cluster2 mirror image_33 snap 2024-04-13T13:38:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:38:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap 2024-04-13T13:38:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-04-13T13:38:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2024-04-13T13:38:27.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2024-04-13T13:38:27.159 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-13T13:38:27.562 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2024-04-13T13:38:27.562 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2024-04-13T13:38:27.563 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2024-04-13T13:38:27.563 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2024-04-13T13:38:27.563 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:38:27.572+0000 7f8114bf8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 33885) UID: 0 2024-04-13T13:38:27.563 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:38:27.572+0000 7f8114bf8640 -1 osd.1 260 *** Got signal Terminated *** 2024-04-13T13:38:27.564 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2024-04-13T13:38:27.572+0000 7f8114bf8640 -1 osd.1 260 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T13:38:27.564 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:38:27.572+0000 7f59263f8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 33888) UID: 0 2024-04-13T13:38:27.564 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:38:27.572+0000 7f59263f8640 -1 osd.2 260 *** Got signal Terminated *** 2024-04-13T13:38:27.564 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2024-04-13T13:38:27.572+0000 7f59263f8640 -1 osd.2 260 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T13:38:27.564 INFO:tasks.ceph.cluster2.mon.a.smithi122.stderr:2024-04-13T13:38:27.572+0000 7f880b298640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 33578) UID: 0 2024-04-13T13:38:27.564 INFO:tasks.ceph.cluster2.mon.a.smithi122.stderr:2024-04-13T13:38:27.572+0000 7f880b298640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-13T13:38:27.565 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:38:27.573+0000 7fa9da5f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 38522) UID: 0 2024-04-13T13:38:27.565 INFO:tasks.ceph.cluster2.mgr.x.smithi083.stderr:2024-04-13T13:38:27.573+0000 7fa9da5f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-13T13:38:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-04-13T13:38:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T13:38:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:38:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2024-04-13T13:38:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2024-04-13T13:38:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2024-04-13T13:38:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-04-13T13:38:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_instance=cluster1 2024-04-13T13:38:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1 2024-04-13T13:38:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ grep -i state 2024-04-13T13:38:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance=cluster1 2024-04-13T13:38:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local loop_instance 2024-04-13T13:38:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-13T13:38:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sleep 0 2024-04-13T13:38:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-13T13:38:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ seq 0 3 2024-04-13T13:38:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:0 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:0 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=0 2024-04-13T13:38:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=0 2024-04-13T13:38:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-13T13:38:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-13T13:38:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-13T13:38:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-13T13:38:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:38:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T13:38:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-13T13:38:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=1 2024-04-13T13:38:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=1 2024-04-13T13:38:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-13T13:38:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-13T13:38:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-13T13:38:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-13T13:38:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2024-04-13T13:38:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2024-04-13T13:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T13:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-13T13:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T13:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ shift 2024-04-13T13:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T13:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T13:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T13:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local local_cluster=cluster1:2 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster=cluster1 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local val=cluster1:2 2024-04-13T13:38:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local cluster_var_name=local_cluster 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ local instance_var_name=instance 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval local_cluster=cluster1 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ local_cluster=cluster1 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ eval instance=2 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ instance=2 2024-04-13T13:38:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-13T13:38:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:38:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:38:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-13T13:38:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-13T13:38:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ return 0 2024-04-13T13:38:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result=' "state": "Replaying"' 2024-04-13T13:38:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo ' "state": "Replaying"' 2024-04-13T13:38:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2024-04-13T13:38:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2024-04-13T13:38:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2024-04-13T13:38:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = journal ']' 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' snapshot = snapshot ']' 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:38:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster2-mirror-image_33.status 2024-04-13T13:38:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster2-mirror-image_33.status 2024-04-13T13:38:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:38:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.6pyizPAJcx/cluster2-mirror-image_33.status 2024-04-13T13:38:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ mkfname cluster1-mirror-image_33.status 2024-04-13T13:38:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo cluster1-mirror-image_33.status 2024-04-13T13:38:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-13T13:38:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_status_log=/tmp/tmp.6pyizPAJcx/cluster1-mirror-image_33.status 2024-04-13T13:38:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-04-13T13:38:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T13:38:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2024-04-13T13:38:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=image_33 2024-04-13T13:38:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-04-13T13:38:29.738 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2669362924135944) > weight (0.25) 2024-04-13T13:38:29.738 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9473500029099271) > weight (0.25) 2024-04-13T13:38:29.738 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6533557949237653) > weight (0.25) 2024-04-13T13:38:29.738 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5631153194883802) > weight (0.25) 2024-04-13T13:38:29.741 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-04-13T13:38:36.848 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.47954083539878356) > weight (0.25) 2024-04-13T13:38:36.848 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-13T13:38:36.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-13T13:38:36.849 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-13T13:38:36.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:2024-04-13T13:38:36.859+0000 7f51ff7f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 56502) UID: 1000 2024-04-13T13:38:38.850 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-13T13:38:38.850 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:39:45.742 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-13T13:39:45.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-13T13:39:45.743 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-13T13:39:45.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr:2024-04-13T13:39:45.753+0000 7f96b63f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 58705) UID: 1000 2024-04-13T13:39:54.744 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-13T13:39:54.745 INFO:teuthology.orchestra.run:waiting for 600 2024-04-13T13:43:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T13:43:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cleanup 110 2024-04-13T13:43:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code=110 2024-04-13T13:43:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2024-04-13T13:43:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 110 -ne 0 ']' 2024-04-13T13:43:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status 2024-04-13T13:43:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster daemon image_pool image_ns image 2024-04-13T13:43:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-04-13T13:43:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 status 2024-04-13T13:43:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 status' 2024-04-13T13:43:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster1 -s 2024-04-13T13:43:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: cluster: 2024-04-13T13:43:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: ba1aae70-25a6-4b6d-86eb-89f270bb7c30 2024-04-13T13:43:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: HEALTH_OK 2024-04-13T13:43:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: services: 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mon: 1 daemons, quorum a (age 31m) 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mgr: x(active, since 31m) 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: osd: 3 osds: 3 up (since 31m), 3 in (since 31m) 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: rbd-mirror: 4 daemons active (1 hosts) 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data: 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pools: 4 pools, 137 pgs 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: objects: 1.21k objects, 4.0 GiB 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: usage: 8.2 GiB used, 262 GiB / 270 GiB avail 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pgs: 137 active+clean 2024-04-13T13:43:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: io: 2024-04-13T13:43:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client: 4.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s rd, 2 op/s wr 2024-04-13T13:43:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster1 service dump 2024-04-13T13:43:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2024-04-13T13:43:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "epoch": 30, 2024-04-13T13:43:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "modified": "2024-04-13T13:42:54.712773+0000", 2024-04-13T13:43:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "services": { 2024-04-13T13:43:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "rbd-mirror": { 2024-04-13T13:43:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "daemons": { 2024-04-13T13:43:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "summary": "", 2024-04-13T13:43:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4317": { 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 6, 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2024-04-13T13:14:35.750296+0000", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4317, 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/3468639170", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "reef", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "18.2.2-1187-gd540ebac", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "centos", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "CentOS Stream 9", 2024-04-13T13:43:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "9", 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.3", 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4317", 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024", 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "5.14.0-435.el9.x86_64", 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "0", 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32488124", 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4843": { 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 26, 2024-04-13T13:43:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2024-04-13T13:34:25.918184+0000", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4843, 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/1557586087", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "reef", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "18.2.2-1187-gd540ebac", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "centos", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "CentOS Stream 9", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "9", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.2", 2024-04-13T13:43:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4843", 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024", 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "5.14.0-435.el9.x86_64", 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "0", 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32488124", 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "5033": { 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 27, 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2024-04-13T13:35:47.949214+0000", 2024-04-13T13:43:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 5033, 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/2508860163", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "reef", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "18.2.2-1187-gd540ebac", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "centos", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "CentOS Stream 9", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "9", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.1", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "5033", 2024-04-13T13:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024", 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "5.14.0-435.el9.x86_64", 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "0", 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32488124", 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "5189": { 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 29, 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2024-04-13T13:36:58.930875+0000", 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 5189, 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/436823691", 2024-04-13T13:43:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "reef", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "18.2.2-1187-gd540ebac", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "centos", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "CentOS Stream 9", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "9", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.0", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "5189", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "5.14.0-435.el9.x86_64", 2024-04-13T13:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "0", 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32488124", 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2024-04-13T13:43:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster1 service status 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "rbd-mirror": { 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4317": { 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2024-04-13T13:43:10.788632+0000", 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2024-04-13T13:43:25.789284+0000", 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":13,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4320\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0,\"instance_id\":\"4322\",\"leader\":true}}" 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4843": { 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2024-04-13T13:43:00.939479+0000", 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2024-04-13T13:43:25.940576+0000", 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":13,\"image_error_count\":0,\"image_local_count\":50,\"image_remote_count\":50,\"image_warning_count\":0,\"instance_id\":\"4846\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4849\"}}" 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "5033": { 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2024-04-13T13:43:22.967830+0000", 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2024-04-13T13:43:27.968058+0000", 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":12,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5036\"},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5039\"}}" 2024-04-13T13:43:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "5189": { 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2024-04-13T13:43:03.947774+0000", 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2024-04-13T13:43:28.948774+0000", 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":12,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5192\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5194\"}}" 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2024-04-13T13:43:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2024-04-13T13:43:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-04-13T13:43:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T13:43:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror images 2024-04-13T13:43:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror images' 2024-04-13T13:43:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1 128 MiB 2 excl 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1@snap 128 MiB 2 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10 128 MiB 2 excl 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10@snap 128 MiB 2 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_11 128 MiB 2 excl 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_11@snap 128 MiB 2 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12 128 MiB 2 excl 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12@snap 128 MiB 2 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_13 128 MiB 2 excl 2024-04-13T13:43:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_13@snap 128 MiB 2 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_14 128 MiB 2 excl 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_14@snap 128 MiB 2 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_15 128 MiB 2 excl 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_15@snap 128 MiB 2 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_16 128 MiB 2 excl 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_16@snap 128 MiB 2 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_17 128 MiB 2 excl 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_17@snap 128 MiB 2 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_18 128 MiB 2 excl 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_18@snap 128 MiB 2 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_19 128 MiB 2 excl 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_19@snap 128 MiB 2 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_2 128 MiB 2 excl 2024-04-13T13:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_2@snap 128 MiB 2 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_20 128 MiB 2 excl 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_20@snap 128 MiB 2 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_21 128 MiB 2 excl 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_21@snap 128 MiB 2 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_22 128 MiB 2 excl 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_22@snap 128 MiB 2 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_23 128 MiB 2 excl 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_23@snap 128 MiB 2 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_24 128 MiB 2 excl 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_24@snap 128 MiB 2 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_25 128 MiB 2 excl 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_25@snap 128 MiB 2 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_26 128 MiB 2 excl 2024-04-13T13:43:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_26@snap 128 MiB 2 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_27 128 MiB 2 excl 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_27@snap 128 MiB 2 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_28 128 MiB 2 excl 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_28@snap 128 MiB 2 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_29 128 MiB 2 excl 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_29@snap 128 MiB 2 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_3 128 MiB 2 excl 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_3@snap 128 MiB 2 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_30 128 MiB 2 excl 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_30@snap 128 MiB 2 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_31 128 MiB 2 excl 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_31@snap 128 MiB 2 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_32 128 MiB 2 excl 2024-04-13T13:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_32@snap 128 MiB 2 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_33 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_34 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_35 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_36 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_37 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_38 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_39 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_4 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_4@snap 128 MiB 2 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_40 128 MiB 2 excl 2024-04-13T13:43:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_41 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_42 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_43 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_44 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_45 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_46 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_47 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_48 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_49 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_5 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_5@snap 128 MiB 2 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_50 128 MiB 2 excl 2024-04-13T13:43:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_6 128 MiB 2 excl 2024-04-13T13:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_6@snap 128 MiB 2 2024-04-13T13:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_7 128 MiB 2 excl 2024-04-13T13:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_7@snap 128 MiB 2 2024-04-13T13:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_8 128 MiB 2 excl 2024-04-13T13:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_8@snap 128 MiB 2 2024-04-13T13:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_9 128 MiB 2 excl 2024-04-13T13:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_9@snap 128 MiB 2 2024-04-13T13:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror mirror pool info 2024-04-13T13:43:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror mirror pool info' 2024-04-13T13:43:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2024-04-13T13:43:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2024-04-13T13:43:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster1 2024-04-13T13:43:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2024-04-13T13:43:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: a2d8dee1-b6c0-44ee-9be5-600f2a9f2571 2024-04-13T13:43:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster2 2024-04-13T13:43:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: 702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2024-04-13T13:43:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2024-04-13T13:43:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror mirror pool status 2024-04-13T13:43:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror mirror pool status' 2024-04-13T13:43:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: WARNING 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: WARNING 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 50 total 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 50 starting_replay 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4317: 2024-04-13T13:43:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4320 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4843: 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4846 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 5033: 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 5036 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 5189: 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 5192 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1: 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015265,"remote_snapshot_timestamp":1713015265,"replay_state":"idle"} 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10: 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 7cb77a9f-0f4a-4ba9-bc4b-ae0678719819 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015336,"remote_snapshot_timestamp":1713015336,"replay_state":"idle"} 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_11: 2024-04-13T13:43:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 59ed4044-a6c7-40ba-a8ff-0113c71c89da 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015343,"remote_snapshot_timestamp":1713015343,"replay_state":"idle"} 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12: 2024-04-13T13:43:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 5b84301a-f350-4908-84da-ddb3ac371259 2024-04-13T13:43:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133856256.0,"last_snapshot_bytes":133856256,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015350,"remote_snapshot_timestamp":1713015350,"replay_state":"idle"} 2024-04-13T13:43:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_13: 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: bf426828-2e69-4ba4-a59f-e3526fdc760d 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015357,"remote_snapshot_timestamp":1713015357,"replay_state":"idle"} 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_14: 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: dc2077e0-ec0a-4427-ab08-fc9e2fa06c71 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134003712.0,"last_snapshot_bytes":134003712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015364,"remote_snapshot_timestamp":1713015364,"replay_state":"idle"} 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_15: 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: f38ae892-421e-4484-b8f6-de0cb55a7833 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015371,"remote_snapshot_timestamp":1713015371,"replay_state":"idle"} 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_16: 2024-04-13T13:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 53de06e5-2f50-4532-ba7a-3aa9dd41bcfd 2024-04-13T13:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133929984.0,"last_snapshot_bytes":133929984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015382,"remote_snapshot_timestamp":1713015382,"replay_state":"idle"} 2024-04-13T13:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_17: 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: d2b64f41-9de5-4eab-8f44-349912e93682 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133715968.0,"last_snapshot_bytes":133715968,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015390,"remote_snapshot_timestamp":1713015390,"replay_state":"idle"} 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_18: 2024-04-13T13:43:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 807d8287-a59d-4634-8f59-0370463ec872 2024-04-13T13:43:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":132408320.0,"last_snapshot_bytes":132408320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015398,"remote_snapshot_timestamp":1713015398,"replay_state":"idle"} 2024-04-13T13:43:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_19: 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 0c18f79e-f883-47a8-84ef-a7789e73b4ba 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133760000.0,"last_snapshot_bytes":133760000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015405,"remote_snapshot_timestamp":1713015405,"replay_state":"idle"} 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_2: 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 45670160-3031-494c-bc36-3e5d858b75c1 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015273,"remote_snapshot_timestamp":1713015273,"replay_state":"idle"} 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_20: 2024-04-13T13:43:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: c3ae73db-b27b-4d07-9b28-0323537a736a 2024-04-13T13:43:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015413,"remote_snapshot_timestamp":1713015413,"replay_state":"idle"} 2024-04-13T13:43:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_21: 2024-04-13T13:43:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 213e24cb-89ef-475f-84d5-92800a3d1efb 2024-04-13T13:43:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133800960.0,"last_snapshot_bytes":133800960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015420,"remote_snapshot_timestamp":1713015420,"replay_state":"idle"} 2024-04-13T13:43:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_22: 2024-04-13T13:43:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: e58af553-340d-46ba-a87f-856f475da231 2024-04-13T13:43:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133537792.0,"last_snapshot_bytes":133537792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015427,"remote_snapshot_timestamp":1713015427,"replay_state":"idle"} 2024-04-13T13:43:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_23: 2024-04-13T13:43:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 8c509f7e-2bfc-4798-88c9-e8a036e7a227 2024-04-13T13:43:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134184960.0,"last_snapshot_bytes":134184960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015434,"remote_snapshot_timestamp":1713015434,"replay_state":"idle"} 2024-04-13T13:43:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_24: 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 91c5558e-79d7-4410-a35b-b56664447d57 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133402624.0,"last_snapshot_bytes":133402624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015441,"remote_snapshot_timestamp":1713015441,"replay_state":"idle"} 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_25: 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 9c3874e8-22df-4fdb-a27a-7e2740eed944 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":131652608.0,"last_snapshot_bytes":131652608,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015448,"remote_snapshot_timestamp":1713015448,"replay_state":"idle"} 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_26: 2024-04-13T13:43:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: cc11b282-5f39-432a-9e34-fd7d24025087 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":131986432.0,"last_snapshot_bytes":131986432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015455,"remote_snapshot_timestamp":1713015455,"replay_state":"idle"} 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_27: 2024-04-13T13:43:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 7a6330ae-57ed-417b-a82d-4d987787e23b 2024-04-13T13:43:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133367808.0,"last_snapshot_bytes":133367808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015462,"remote_snapshot_timestamp":1713015462,"replay_state":"idle"} 2024-04-13T13:43:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_28: 2024-04-13T13:43:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 6c4ef21c-2a7c-465b-b23d-96050d4d8338 2024-04-13T13:43:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":132506624.0,"last_snapshot_bytes":132506624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015470,"remote_snapshot_timestamp":1713015470,"replay_state":"idle"} 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_29: 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: d1a11424-da29-4237-abbf-b187574ee6bd 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133400576.0,"last_snapshot_bytes":133400576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015478,"remote_snapshot_timestamp":1713015478,"replay_state":"idle"} 2024-04-13T13:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_3: 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 173021c1-45fc-4ddf-ae55-e453164e52a8 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015281,"remote_snapshot_timestamp":1713015281,"replay_state":"idle"} 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_30: 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: df429b8a-ec65-47e3-b367-41f75bbe4da1 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133539840.0,"last_snapshot_bytes":133539840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015486,"remote_snapshot_timestamp":1713015486,"replay_state":"idle"} 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_31: 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: f666e830-e8d8-4600-a27f-f8027ab46f85 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134102016.0,"last_snapshot_bytes":134102016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015494,"remote_snapshot_timestamp":1713015494,"replay_state":"idle"} 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_32: 2024-04-13T13:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: c95975b5-0eee-4f76-8be8-10a90b792c79 2024-04-13T13:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133872640.0,"last_snapshot_bytes":133872640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015501,"remote_snapshot_timestamp":1713015501,"replay_state":"idle"} 2024-04-13T13:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_33: 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 51d2605d-d07c-452b-98c9-76a5433b1837 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015021,"remote_snapshot_timestamp":1713015021,"replay_state":"idle"} 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_34: 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 1f7e0537-71e2-4dcf-a741-47e1160880d2 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015030,"remote_snapshot_timestamp":1713015030,"replay_state":"idle"} 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_35: 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 1dd9564b-377b-4a17-82b5-1bee324ef4f2 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015036,"remote_snapshot_timestamp":1713015036,"replay_state":"idle"} 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_36: 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 37c03058-f2ca-48b7-b47d-ccfcb255694b 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015044,"remote_snapshot_timestamp":1713015044,"replay_state":"idle"} 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_37: 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: b1d1e9b9-23ac-4857-a6a7-e354f28ab5cd 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015057,"remote_snapshot_timestamp":1713015057,"replay_state":"idle"} 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_38: 2024-04-13T13:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: d96b2a28-af93-4496-b387-393648c236e0 2024-04-13T13:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015071,"remote_snapshot_timestamp":1713015071,"replay_state":"idle"} 2024-04-13T13:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_39: 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 41ea30c5-3df3-4041-8289-68ecbae98d63 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015086,"remote_snapshot_timestamp":1713015086,"replay_state":"idle"} 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_4: 2024-04-13T13:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 8d69fb4b-4707-45d2-bd35-f5dc2bafa56b 2024-04-13T13:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015291,"remote_snapshot_timestamp":1713015291,"replay_state":"idle"} 2024-04-13T13:43:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_40: 2024-04-13T13:43:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 8ed18fcc-ddf5-45b3-bceb-3cbe6226dcaa 2024-04-13T13:43:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015099,"remote_snapshot_timestamp":1713015099,"replay_state":"idle"} 2024-04-13T13:43:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_41: 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 682df14d-9783-44dd-8781-b0f0d506d592 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015113,"remote_snapshot_timestamp":1713015113,"replay_state":"idle"} 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_42: 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: a05e5ef4-930c-4ff6-b504-51ec542751fd 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015125,"remote_snapshot_timestamp":1713015125,"replay_state":"idle"} 2024-04-13T13:43:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_43: 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: c0e8ec4e-880c-4448-9702-ba9518d9f196 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015138,"remote_snapshot_timestamp":1713015138,"replay_state":"idle"} 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_44: 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 8f12f1ec-dc11-465e-a774-65eca20bf996 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015151,"remote_snapshot_timestamp":1713015151,"replay_state":"idle"} 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_45: 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 4722198d-f141-466b-aec9-9f1bd91cfc3f 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015165,"remote_snapshot_timestamp":1713015165,"replay_state":"idle"} 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_46: 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: fe7f2378-9fe8-4c98-84bb-5958971e249f 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015171,"remote_snapshot_timestamp":1713015171,"replay_state":"idle"} 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_47: 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 1e77a3c2-50cb-4ed3-be13-ff3d35f16c46 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015184,"remote_snapshot_timestamp":1713015184,"replay_state":"idle"} 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_48: 2024-04-13T13:43:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: fee9868f-6c82-4814-a3f0-2803aaff14e9 2024-04-13T13:43:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015192,"remote_snapshot_timestamp":1713015192,"replay_state":"idle"} 2024-04-13T13:43:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_49: 2024-04-13T13:43:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 89d28a1c-d65d-4fb9-b8cc-35062079a2b1 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015206,"remote_snapshot_timestamp":1713015206,"replay_state":"idle"} 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_5: 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: b72ab703-5a35-4d41-90bc-db2c081ebc57 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015299,"remote_snapshot_timestamp":1713015299,"replay_state":"idle"} 2024-04-13T13:43:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_50: 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: bb8a902e-d932-412f-bc77-def5842a6663 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015217,"remote_snapshot_timestamp":1713015217,"replay_state":"idle"} 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:05 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:42:59 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_6: 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 1466461b-b5d7-4bb3-9928-7a3a2fd5b22f 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015307,"remote_snapshot_timestamp":1713015307,"replay_state":"idle"} 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:06 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_7: 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 5e948815-4bc4-4300-914c-1855c84e9f33 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015315,"remote_snapshot_timestamp":1713015315,"replay_state":"idle"} 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:25 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_8: 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 970b3a1e-6448-44f1-be2c-e3b2bc2b08af 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015322,"remote_snapshot_timestamp":1713015322,"replay_state":"idle"} 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:18 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_9: 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 9b0f1c92-41a8-4559-8a23-3818c789553a 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713015329,"remote_snapshot_timestamp":1713015329,"replay_state":"idle"} 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:28 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+starting_replay 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: starting replay 2024-04-13T13:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2024-04-13 13:43:17 2024-04-13T13:43:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls 2024-04-13T13:43:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_1 info 2024-04-13T13:43:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_1 info' 2024-04-13T13:43:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_1 2024-04-13T13:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_1': 2024-04-13T13:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab9e4a96b9 2024-04-13T13:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab9e4a96b9 2024-04-13T13:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:13 2024 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:34:29 2024 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:13 2024 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f 2024-04-13T13:43:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_1 journal status 2024-04-13T13:43:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_1 journal status' 2024-04-13T13:43:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_1 2024-04-13T13:43:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_1 2024-04-13T13:43:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_1 snapshots 2024-04-13T13:43:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_1 snapshots' 2024-04-13T13:43:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_1 2024-04-13T13:43:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 124 snap 128 MiB Sat Apr 13 13:34:26 2024 user 2024-04-13T13:43:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 125 .mirror.non_primary.bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f.a64267e9-3993-4475-9049-808497f3c204 128 MiB Sat Apr 13 13:34:27 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:119 copied) 2024-04-13T13:43:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_10 info 2024-04-13T13:43:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_10 info' 2024-04-13T13:43:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_10 2024-04-13T13:43:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_10': 2024-04-13T13:43:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0e34d4a0e 2024-04-13T13:43:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0e34d4a0e 2024-04-13T13:43:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:56 2024 2024-04-13T13:43:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:35:39 2024 2024-04-13T13:43:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:56 2024 2024-04-13T13:43:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 7cb77a9f-0f4a-4ba9-bc4b-ae0678719819 2024-04-13T13:43:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_10 journal status 2024-04-13T13:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_10 journal status' 2024-04-13T13:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_10 2024-04-13T13:43:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_10 2024-04-13T13:43:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_10 snapshots 2024-04-13T13:43:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_10 snapshots' 2024-04-13T13:43:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_10 2024-04-13T13:43:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 152 snap 128 MiB Sat Apr 13 13:35:37 2024 user 2024-04-13T13:43:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 153 .mirror.non_primary.7cb77a9f-0f4a-4ba9-bc4b-ae0678719819.0ecdb140-cd2a-456c-8d42-3bdf401d3ab7 128 MiB Sat Apr 13 13:35:38 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:137 copied) 2024-04-13T13:43:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_11 info 2024-04-13T13:43:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_11 info' 2024-04-13T13:43:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_11 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_11': 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c21fe77c7 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c21fe77c7 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:04 2024 2024-04-13T13:43:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:35:46 2024 2024-04-13T13:43:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:04 2024 2024-04-13T13:43:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 59ed4044-a6c7-40ba-a8ff-0113c71c89da 2024-04-13T13:43:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_11 journal status 2024-04-13T13:43:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_11 journal status' 2024-04-13T13:43:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_11 2024-04-13T13:43:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_11 2024-04-13T13:43:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_11 snapshots 2024-04-13T13:43:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_11 snapshots' 2024-04-13T13:43:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_11 2024-04-13T13:43:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 155 snap 128 MiB Sat Apr 13 13:35:44 2024 user 2024-04-13T13:43:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 156 .mirror.non_primary.59ed4044-a6c7-40ba-a8ff-0113c71c89da.ca62b980-6a2c-4809-8ac3-58c11fc5f291 128 MiB Sat Apr 13 13:35:45 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:139 copied) 2024-04-13T13:43:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_12 info 2024-04-13T13:43:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_12 info' 2024-04-13T13:43:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_12 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_12': 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b3bdf2e083 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b3bdf2e083 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:10 2024 2024-04-13T13:43:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:35:53 2024 2024-04-13T13:43:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:10 2024 2024-04-13T13:43:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 5b84301a-f350-4908-84da-ddb3ac371259 2024-04-13T13:43:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_12 journal status 2024-04-13T13:43:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_12 journal status' 2024-04-13T13:43:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_12 2024-04-13T13:43:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_12 2024-04-13T13:43:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_12 snapshots 2024-04-13T13:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_12 snapshots' 2024-04-13T13:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_12 2024-04-13T13:43:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 158 snap 128 MiB Sat Apr 13 13:35:51 2024 user 2024-04-13T13:43:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 159 .mirror.non_primary.5b84301a-f350-4908-84da-ddb3ac371259.c88c8a7d-1592-4f44-8baf-03cd734f7cab 128 MiB Sat Apr 13 13:35:52 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:141 copied) 2024-04-13T13:43:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_13 info 2024-04-13T13:43:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_13 info' 2024-04-13T13:43:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_13 2024-04-13T13:43:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_13': 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10abb6a353e2 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10abb6a353e2 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:18 2024 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:36:00 2024 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:18 2024 2024-04-13T13:43:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: bf426828-2e69-4ba4-a59f-e3526fdc760d 2024-04-13T13:43:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_13 journal status 2024-04-13T13:43:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_13 journal status' 2024-04-13T13:43:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_13 2024-04-13T13:43:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_13 2024-04-13T13:43:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_13 snapshots' 2024-04-13T13:43:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_13 snapshots 2024-04-13T13:43:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_13 2024-04-13T13:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 161 snap 128 MiB Sat Apr 13 13:35:58 2024 user 2024-04-13T13:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 162 .mirror.non_primary.bf426828-2e69-4ba4-a59f-e3526fdc760d.bafbfdb4-072a-4374-912c-646f5560dfb6 128 MiB Sat Apr 13 13:35:59 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:143 copied) 2024-04-13T13:43:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_14 info 2024-04-13T13:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_14 info' 2024-04-13T13:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_14 2024-04-13T13:43:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_14': 2024-04-13T13:43:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0ecc4435e 2024-04-13T13:43:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0ecc4435e 2024-04-13T13:43:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:30 2024 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:36:07 2024 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:30 2024 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: dc2077e0-ec0a-4427-ab08-fc9e2fa06c71 2024-04-13T13:43:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_14 journal status 2024-04-13T13:43:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_14 journal status' 2024-04-13T13:43:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_14 2024-04-13T13:43:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_14 2024-04-13T13:43:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_14 snapshots 2024-04-13T13:43:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_14 snapshots' 2024-04-13T13:43:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_14 2024-04-13T13:43:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 164 snap 128 MiB Sat Apr 13 13:36:04 2024 user 2024-04-13T13:43:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 165 .mirror.non_primary.dc2077e0-ec0a-4427-ab08-fc9e2fa06c71.39f91607-e2f2-418b-8a74-b79e8c03c39f 128 MiB Sat Apr 13 13:36:05 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:145 copied) 2024-04-13T13:43:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_15 info 2024-04-13T13:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_15 info' 2024-04-13T13:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_15 2024-04-13T13:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_15': 2024-04-13T13:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124cdadfabd0 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124cdadfabd0 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:42 2024 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:36:14 2024 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:42 2024 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: f38ae892-421e-4484-b8f6-de0cb55a7833 2024-04-13T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_15 journal status 2024-04-13T13:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_15 journal status' 2024-04-13T13:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_15 2024-04-13T13:43:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_15 2024-04-13T13:43:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_15 snapshots 2024-04-13T13:43:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_15 snapshots' 2024-04-13T13:43:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_15 2024-04-13T13:43:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 167 snap 128 MiB Sat Apr 13 13:36:12 2024 user 2024-04-13T13:43:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 168 .mirror.non_primary.f38ae892-421e-4484-b8f6-de0cb55a7833.e33c039c-9ef3-4dfe-b5b4-936745934aff 128 MiB Sat Apr 13 13:36:12 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:147 copied) 2024-04-13T13:43:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_16 info 2024-04-13T13:43:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_16 info' 2024-04-13T13:43:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_16 2024-04-13T13:43:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_16': 2024-04-13T13:43:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b3f0d72789 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b3f0d72789 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:49 2024 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:36:26 2024 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:49 2024 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 53de06e5-2f50-4532-ba7a-3aa9dd41bcfd 2024-04-13T13:43:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_16 journal status 2024-04-13T13:43:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_16 journal status' 2024-04-13T13:43:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_16 2024-04-13T13:43:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_16 2024-04-13T13:43:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_16 snapshots 2024-04-13T13:43:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_16 snapshots' 2024-04-13T13:43:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_16 2024-04-13T13:43:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 170 snap 128 MiB Sat Apr 13 13:36:22 2024 user 2024-04-13T13:43:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 171 .mirror.non_primary.53de06e5-2f50-4532-ba7a-3aa9dd41bcfd.5a558c17-ffca-4c7f-85d9-6853fdd3b0d2 128 MiB Sat Apr 13 13:36:23 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:149 copied) 2024-04-13T13:43:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_17 info 2024-04-13T13:43:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_17 info' 2024-04-13T13:43:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_17 2024-04-13T13:43:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_17': 2024-04-13T13:43:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab656b9041 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab656b9041 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:27:59 2024 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:36:34 2024 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:27:59 2024 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: d2b64f41-9de5-4eab-8f44-349912e93682 2024-04-13T13:43:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_17 journal status 2024-04-13T13:43:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_17 journal status' 2024-04-13T13:43:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_17 2024-04-13T13:43:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_17 2024-04-13T13:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_17 snapshots 2024-04-13T13:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_17 snapshots' 2024-04-13T13:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_17 2024-04-13T13:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 173 snap 128 MiB Sat Apr 13 13:36:31 2024 user 2024-04-13T13:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 174 .mirror.non_primary.d2b64f41-9de5-4eab-8f44-349912e93682.d54d5d62-68f4-42dd-aac1-5ffcf5d4a72a 128 MiB Sat Apr 13 13:36:32 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:151 copied) 2024-04-13T13:43:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_18 info 2024-04-13T13:43:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_18 info' 2024-04-13T13:43:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_18 2024-04-13T13:43:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_18': 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0daec96fa 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0daec96fa 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:11 2024 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:36:41 2024 2024-04-13T13:43:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:11 2024 2024-04-13T13:43:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 807d8287-a59d-4634-8f59-0370463ec872 2024-04-13T13:43:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_18 journal status 2024-04-13T13:43:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_18 journal status' 2024-04-13T13:43:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_18 2024-04-13T13:43:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_18 2024-04-13T13:43:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_18 snapshots 2024-04-13T13:43:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_18 snapshots' 2024-04-13T13:43:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_18 2024-04-13T13:43:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 176 snap 128 MiB Sat Apr 13 13:36:39 2024 user 2024-04-13T13:43:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 177 .mirror.non_primary.807d8287-a59d-4634-8f59-0370463ec872.471cb8df-5aa2-4fb7-a2ab-4ba68e674ae3 128 MiB Sat Apr 13 13:36:40 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:153 copied) 2024-04-13T13:43:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_19 info 2024-04-13T13:43:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_19 info' 2024-04-13T13:43:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_19 2024-04-13T13:43:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_19': 2024-04-13T13:43:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124ced325f9d 2024-04-13T13:43:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124ced325f9d 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:18 2024 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:36:50 2024 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:18 2024 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 0c18f79e-f883-47a8-84ef-a7789e73b4ba 2024-04-13T13:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_19 journal status 2024-04-13T13:43:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_19 journal status' 2024-04-13T13:43:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_19 2024-04-13T13:43:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_19 2024-04-13T13:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_19 snapshots 2024-04-13T13:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_19 snapshots' 2024-04-13T13:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_19 2024-04-13T13:43:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 179 snap 128 MiB Sat Apr 13 13:36:46 2024 user 2024-04-13T13:43:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 180 .mirror.non_primary.0c18f79e-f883-47a8-84ef-a7789e73b4ba.c3cbec2f-5108-452d-afe0-c04781e640dc 128 MiB Sat Apr 13 13:36:47 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:155 copied) 2024-04-13T13:43:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_2 info 2024-04-13T13:43:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_2 info' 2024-04-13T13:43:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_2 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_2': 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0d185c1 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0d185c1 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:25 2024 2024-04-13T13:43:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:34:37 2024 2024-04-13T13:43:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:25 2024 2024-04-13T13:43:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 45670160-3031-494c-bc36-3e5d858b75c1 2024-04-13T13:43:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_2 journal status 2024-04-13T13:43:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_2 journal status' 2024-04-13T13:43:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_2 2024-04-13T13:43:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_2 2024-04-13T13:43:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_2 snapshots 2024-04-13T13:43:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_2 snapshots' 2024-04-13T13:43:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_2 2024-04-13T13:43:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 127 snap 128 MiB Sat Apr 13 13:34:34 2024 user 2024-04-13T13:43:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 128 .mirror.non_primary.45670160-3031-494c-bc36-3e5d858b75c1.577dc98f-3bab-4d57-86dc-fd0d6a701eef 128 MiB Sat Apr 13 13:34:35 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:121 copied) 2024-04-13T13:43:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_20 info' 2024-04-13T13:43:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_20 info 2024-04-13T13:43:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_20 2024-04-13T13:43:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_20': 2024-04-13T13:43:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b39eb8c6ed 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b39eb8c6ed 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:28 2024 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:36:56 2024 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:28 2024 2024-04-13T13:43:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: c3ae73db-b27b-4d07-9b28-0323537a736a 2024-04-13T13:43:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_20 journal status 2024-04-13T13:43:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_20 journal status' 2024-04-13T13:43:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_20 2024-04-13T13:43:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_20 2024-04-13T13:43:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_20 snapshots 2024-04-13T13:43:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_20 snapshots' 2024-04-13T13:43:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_20 2024-04-13T13:43:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 182 snap 128 MiB Sat Apr 13 13:36:54 2024 user 2024-04-13T13:43:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 183 .mirror.non_primary.c3ae73db-b27b-4d07-9b28-0323537a736a.3d4a9975-e5b9-48c5-a1b0-6a2b1a342494 128 MiB Sat Apr 13 13:36:55 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:157 copied) 2024-04-13T13:43:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_21 info 2024-04-13T13:43:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_21 info' 2024-04-13T13:43:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_21 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_21': 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10abeb6b2184 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10abeb6b2184 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:33 2024 2024-04-13T13:43:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:37:03 2024 2024-04-13T13:43:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:33 2024 2024-04-13T13:43:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 213e24cb-89ef-475f-84d5-92800a3d1efb 2024-04-13T13:43:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_21 journal status 2024-04-13T13:43:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_21 journal status' 2024-04-13T13:43:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_21 2024-04-13T13:43:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_21 2024-04-13T13:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_21 snapshots 2024-04-13T13:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_21 snapshots' 2024-04-13T13:43:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_21 2024-04-13T13:43:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 185 snap 128 MiB Sat Apr 13 13:37:01 2024 user 2024-04-13T13:43:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 186 .mirror.non_primary.213e24cb-89ef-475f-84d5-92800a3d1efb.7480ceea-18d4-4f51-b937-e426e055b440 128 MiB Sat Apr 13 13:37:02 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:159 copied) 2024-04-13T13:43:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_22 info 2024-04-13T13:43:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_22 info' 2024-04-13T13:43:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_22 2024-04-13T13:43:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_22': 2024-04-13T13:43:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e03f1c9867 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e03f1c9867 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:45 2024 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:37:10 2024 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:45 2024 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: e58af553-340d-46ba-a87f-856f475da231 2024-04-13T13:43:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_22 journal status 2024-04-13T13:43:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_22 journal status' 2024-04-13T13:43:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_22 2024-04-13T13:43:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_22 2024-04-13T13:43:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_22 snapshots 2024-04-13T13:43:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_22 snapshots' 2024-04-13T13:43:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_22 2024-04-13T13:43:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 188 snap 128 MiB Sat Apr 13 13:37:08 2024 user 2024-04-13T13:43:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 189 .mirror.non_primary.e58af553-340d-46ba-a87f-856f475da231.9c2aac79-8ae4-4e1a-80ae-480ed7d686f5 128 MiB Sat Apr 13 13:37:09 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:161 copied) 2024-04-13T13:43:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_23 info 2024-04-13T13:43:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_23 info' 2024-04-13T13:43:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_23 2024-04-13T13:43:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_23': 2024-04-13T13:43:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c1a2d37a 2024-04-13T13:43:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c1a2d37a 2024-04-13T13:43:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:28:52 2024 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:37:17 2024 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:28:52 2024 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8c509f7e-2bfc-4798-88c9-e8a036e7a227 2024-04-13T13:43:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_23 journal status 2024-04-13T13:43:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_23 journal status' 2024-04-13T13:43:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_23 2024-04-13T13:43:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_23 2024-04-13T13:43:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_23 snapshots 2024-04-13T13:43:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_23 snapshots' 2024-04-13T13:43:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_23 2024-04-13T13:43:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 191 snap 128 MiB Sat Apr 13 13:37:15 2024 user 2024-04-13T13:43:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 192 .mirror.non_primary.8c509f7e-2bfc-4798-88c9-e8a036e7a227.5c31862b-6368-463f-a0dd-f11eefd79988 128 MiB Sat Apr 13 13:37:16 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:163 copied) 2024-04-13T13:43:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_24 info 2024-04-13T13:43:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_24 info' 2024-04-13T13:43:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_24 2024-04-13T13:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_24': 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b3ce2403df 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b3ce2403df 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:01 2024 2024-04-13T13:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:37:24 2024 2024-04-13T13:43:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:01 2024 2024-04-13T13:43:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 91c5558e-79d7-4410-a35b-b56664447d57 2024-04-13T13:43:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_24 journal status 2024-04-13T13:43:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_24 journal status' 2024-04-13T13:43:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_24 2024-04-13T13:43:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_24 2024-04-13T13:43:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_24 snapshots 2024-04-13T13:43:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_24 snapshots' 2024-04-13T13:43:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_24 2024-04-13T13:43:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 194 snap 128 MiB Sat Apr 13 13:37:22 2024 user 2024-04-13T13:43:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 195 .mirror.non_primary.91c5558e-79d7-4410-a35b-b56664447d57.53785203-7257-42c1-b59c-454121b31995 128 MiB Sat Apr 13 13:37:23 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:165 copied) 2024-04-13T13:43:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_25 info 2024-04-13T13:43:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_25 info' 2024-04-13T13:43:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_25 2024-04-13T13:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_25': 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10abe9899b25 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10abe9899b25 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:07 2024 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:37:31 2024 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:07 2024 2024-04-13T13:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 9c3874e8-22df-4fdb-a27a-7e2740eed944 2024-04-13T13:43:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_25 journal status 2024-04-13T13:43:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_25 journal status' 2024-04-13T13:43:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_25 2024-04-13T13:43:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_25 2024-04-13T13:43:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_25 snapshots 2024-04-13T13:43:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_25 snapshots' 2024-04-13T13:43:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_25 2024-04-13T13:43:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 197 snap 128 MiB Sat Apr 13 13:37:29 2024 user 2024-04-13T13:43:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 198 .mirror.non_primary.9c3874e8-22df-4fdb-a27a-7e2740eed944.d4d65de1-50bd-484c-8eae-b8993e68bacf 128 MiB Sat Apr 13 13:37:30 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:167 copied) 2024-04-13T13:43:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_26 info 2024-04-13T13:43:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_26 info' 2024-04-13T13:43:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_26 2024-04-13T13:43:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_26': 2024-04-13T13:43:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0270e3823 2024-04-13T13:43:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0270e3823 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:14 2024 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:37:38 2024 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:14 2024 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: cc11b282-5f39-432a-9e34-fd7d24025087 2024-04-13T13:43:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_26 journal status 2024-04-13T13:43:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_26 journal status' 2024-04-13T13:43:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_26 2024-04-13T13:43:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_26 2024-04-13T13:43:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_26 snapshots 2024-04-13T13:43:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_26 snapshots' 2024-04-13T13:43:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_26 2024-04-13T13:43:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 200 snap 128 MiB Sat Apr 13 13:37:36 2024 user 2024-04-13T13:43:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 201 .mirror.non_primary.cc11b282-5f39-432a-9e34-fd7d24025087.6ede890d-3726-4b22-9349-42f041b46874 128 MiB Sat Apr 13 13:37:37 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:169 copied) 2024-04-13T13:43:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_27 info 2024-04-13T13:43:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_27 info' 2024-04-13T13:43:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_27 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_27': 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c4cccc1c3 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c4cccc1c3 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:20 2024 2024-04-13T13:43:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:37:45 2024 2024-04-13T13:43:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:20 2024 2024-04-13T13:43:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 7a6330ae-57ed-417b-a82d-4d987787e23b 2024-04-13T13:43:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:32.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_27 journal status 2024-04-13T13:43:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_27 journal status' 2024-04-13T13:43:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_27 2024-04-13T13:43:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_27 2024-04-13T13:43:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_27 snapshots 2024-04-13T13:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_27 snapshots' 2024-04-13T13:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_27 2024-04-13T13:43:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 203 snap 128 MiB Sat Apr 13 13:37:43 2024 user 2024-04-13T13:43:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 204 .mirror.non_primary.7a6330ae-57ed-417b-a82d-4d987787e23b.333b1f8b-25eb-4354-a1cc-c284c92f3d46 128 MiB Sat Apr 13 13:37:44 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:171 copied) 2024-04-13T13:43:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_28 info 2024-04-13T13:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_28 info' 2024-04-13T13:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_28 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_28': 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b38bced57c 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b38bced57c 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:27 2024 2024-04-13T13:43:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:37:54 2024 2024-04-13T13:43:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:27 2024 2024-04-13T13:43:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 6c4ef21c-2a7c-465b-b23d-96050d4d8338 2024-04-13T13:43:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_28 journal status 2024-04-13T13:43:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_28 journal status' 2024-04-13T13:43:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_28 2024-04-13T13:43:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_28 2024-04-13T13:43:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_28 snapshots 2024-04-13T13:43:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_28 snapshots' 2024-04-13T13:43:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_28 2024-04-13T13:43:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 206 snap 128 MiB Sat Apr 13 13:37:51 2024 user 2024-04-13T13:43:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 207 .mirror.non_primary.6c4ef21c-2a7c-465b-b23d-96050d4d8338.6361eaae-31de-4169-938b-819a6d88b0c8 128 MiB Sat Apr 13 13:37:52 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:173 copied) 2024-04-13T13:43:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_29 info 2024-04-13T13:43:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_29 info' 2024-04-13T13:43:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_29 2024-04-13T13:43:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_29': 2024-04-13T13:43:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10aba9bacfc5 2024-04-13T13:43:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10aba9bacfc5 2024-04-13T13:43:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:39 2024 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:38:02 2024 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:39 2024 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: d1a11424-da29-4237-abbf-b187574ee6bd 2024-04-13T13:43:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_29 journal status 2024-04-13T13:43:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_29 journal status' 2024-04-13T13:43:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_29 2024-04-13T13:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_29 2024-04-13T13:43:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_29 snapshots 2024-04-13T13:43:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_29 snapshots' 2024-04-13T13:43:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_29 2024-04-13T13:43:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 209 snap 128 MiB Sat Apr 13 13:37:59 2024 user 2024-04-13T13:43:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 210 .mirror.non_primary.d1a11424-da29-4237-abbf-b187574ee6bd.bfe0eb17-9c5e-42ed-8956-dcbf641c9aa2 128 MiB Sat Apr 13 13:38:00 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:175 copied) 2024-04-13T13:43:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_3 info 2024-04-13T13:43:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_3 info' 2024-04-13T13:43:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_3 2024-04-13T13:43:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_3': 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124cedb7a54f 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124cedb7a54f 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:43:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:34:47 2024 2024-04-13T13:43:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:43:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 173021c1-45fc-4ddf-ae55-e453164e52a8 2024-04-13T13:43:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_3 journal status 2024-04-13T13:43:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_3 journal status' 2024-04-13T13:43:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_3 2024-04-13T13:43:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_3 2024-04-13T13:43:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_3 snapshots 2024-04-13T13:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_3 snapshots' 2024-04-13T13:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_3 2024-04-13T13:43:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 130 snap 128 MiB Sat Apr 13 13:34:42 2024 user 2024-04-13T13:43:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 132 .mirror.non_primary.173021c1-45fc-4ddf-ae55-e453164e52a8.cd10cc4b-8a76-4e49-a3bd-fcd730349777 128 MiB Sat Apr 13 13:34:44 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:123 copied) 2024-04-13T13:43:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_30 info 2024-04-13T13:43:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_30 info' 2024-04-13T13:43:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_30 2024-04-13T13:43:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_30': 2024-04-13T13:43:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e051dfcca0 2024-04-13T13:43:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e051dfcca0 2024-04-13T13:43:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:29:51 2024 2024-04-13T13:43:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:38:10 2024 2024-04-13T13:43:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:29:51 2024 2024-04-13T13:43:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: df429b8a-ec65-47e3-b367-41f75bbe4da1 2024-04-13T13:43:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_30 journal status 2024-04-13T13:43:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_30 journal status' 2024-04-13T13:43:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_30 2024-04-13T13:43:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_30 2024-04-13T13:43:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_30 snapshots 2024-04-13T13:43:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_30 snapshots' 2024-04-13T13:43:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_30 2024-04-13T13:43:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 212 snap 128 MiB Sat Apr 13 13:38:07 2024 user 2024-04-13T13:43:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 213 .mirror.non_primary.df429b8a-ec65-47e3-b367-41f75bbe4da1.02f1032b-51be-4a57-9484-a2991369cfc1 128 MiB Sat Apr 13 13:38:08 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:177 copied) 2024-04-13T13:43:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_31 info 2024-04-13T13:43:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_31 info' 2024-04-13T13:43:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_31 2024-04-13T13:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_31': 2024-04-13T13:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c2ea13d0c7 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c2ea13d0c7 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:00 2024 2024-04-13T13:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:38:17 2024 2024-04-13T13:43:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:00 2024 2024-04-13T13:43:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: f666e830-e8d8-4600-a27f-f8027ab46f85 2024-04-13T13:43:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_31 journal status 2024-04-13T13:43:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_31 journal status' 2024-04-13T13:43:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_31 2024-04-13T13:43:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_31 2024-04-13T13:43:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_31 snapshots 2024-04-13T13:43:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_31 snapshots' 2024-04-13T13:43:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_31 2024-04-13T13:43:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 215 snap 128 MiB Sat Apr 13 13:38:15 2024 user 2024-04-13T13:43:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 216 .mirror.non_primary.f666e830-e8d8-4600-a27f-f8027ab46f85.32b610cc-edf2-468c-96b4-f02b5845f595 128 MiB Sat Apr 13 13:38:16 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:179 copied) 2024-04-13T13:43:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_32 info 2024-04-13T13:43:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_32 info' 2024-04-13T13:43:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_32 2024-04-13T13:43:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_32': 2024-04-13T13:43:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c2cefedd3b 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c2cefedd3b 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:12 2024 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:38:24 2024 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:12 2024 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: c95975b5-0eee-4f76-8be8-10a90b792c79 2024-04-13T13:43:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_32 journal status 2024-04-13T13:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_32 journal status' 2024-04-13T13:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_32 2024-04-13T13:43:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_32 2024-04-13T13:43:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_32 snapshots 2024-04-13T13:43:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_32 snapshots' 2024-04-13T13:43:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_32 2024-04-13T13:43:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 218 snap 128 MiB Sat Apr 13 13:38:22 2024 user 2024-04-13T13:43:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 219 .mirror.non_primary.c95975b5-0eee-4f76-8be8-10a90b792c79.6a6ff527-64dc-4ce0-8638-55fcb1b77595 128 MiB Sat Apr 13 13:38:23 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:181 copied) 2024-04-13T13:43:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_33 info 2024-04-13T13:43:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_33 info' 2024-04-13T13:43:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_33 2024-04-13T13:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_33': 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab2d1d981 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab2d1d981 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:23 2024 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:23 2024 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:23 2024 2024-04-13T13:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 51d2605d-d07c-452b-98c9-76a5433b1837 2024-04-13T13:43:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_33 journal status 2024-04-13T13:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_33 journal status' 2024-04-13T13:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_33 2024-04-13T13:43:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_33 2024-04-13T13:43:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_33 snapshots 2024-04-13T13:43:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_33 snapshots' 2024-04-13T13:43:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_33 2024-04-13T13:43:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 106 .mirror.non_primary.51d2605d-d07c-452b-98c9-76a5433b1837.9983a617-e614-4e8d-9054-3b7a11e2f8d7 128 MiB Sat Apr 13 13:30:24 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:100 copied) 2024-04-13T13:43:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_34 info 2024-04-13T13:43:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_34 info' 2024-04-13T13:43:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_34 2024-04-13T13:43:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_34': 2024-04-13T13:43:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e056f99e57 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e056f99e57 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:32 2024 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:32 2024 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:32 2024 2024-04-13T13:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1f7e0537-71e2-4dcf-a741-47e1160880d2 2024-04-13T13:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_34 journal status 2024-04-13T13:43:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_34 journal status' 2024-04-13T13:43:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_34 2024-04-13T13:43:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_34 2024-04-13T13:43:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_34 snapshots 2024-04-13T13:43:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_34 snapshots' 2024-04-13T13:43:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_34 2024-04-13T13:43:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 107 .mirror.non_primary.1f7e0537-71e2-4dcf-a741-47e1160880d2.12e7cd13-d5f5-4415-9448-3e283d0c90da 128 MiB Sat Apr 13 13:30:33 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:101 copied) 2024-04-13T13:43:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_35 info 2024-04-13T13:43:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_35 info' 2024-04-13T13:43:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_35 2024-04-13T13:43:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_35': 2024-04-13T13:43:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c28eef337 2024-04-13T13:43:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c28eef337 2024-04-13T13:43:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:38 2024 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:38 2024 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:38 2024 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1dd9564b-377b-4a17-82b5-1bee324ef4f2 2024-04-13T13:43:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_35 journal status 2024-04-13T13:43:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_35 journal status' 2024-04-13T13:43:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_35 2024-04-13T13:43:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_35 2024-04-13T13:43:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_35 snapshots 2024-04-13T13:43:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_35 snapshots' 2024-04-13T13:43:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_35 2024-04-13T13:43:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 108 .mirror.non_primary.1dd9564b-377b-4a17-82b5-1bee324ef4f2.cb6bd9b6-33c1-4ac0-a18f-ac9a6ee09e9a 128 MiB Sat Apr 13 13:30:39 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:102 copied) 2024-04-13T13:43:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_36 info 2024-04-13T13:43:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_36 info' 2024-04-13T13:43:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_36 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_36': 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c2902e777d 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c2902e777d 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:46 2024 2024-04-13T13:43:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:46 2024 2024-04-13T13:43:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:46 2024 2024-04-13T13:43:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 37c03058-f2ca-48b7-b47d-ccfcb255694b 2024-04-13T13:43:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_36 journal status 2024-04-13T13:43:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_36 journal status' 2024-04-13T13:43:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_36 2024-04-13T13:43:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_36 2024-04-13T13:43:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_36 snapshots 2024-04-13T13:43:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_36 snapshots' 2024-04-13T13:43:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_36 2024-04-13T13:43:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 109 .mirror.non_primary.37c03058-f2ca-48b7-b47d-ccfcb255694b.731c42f6-358f-4205-b8b1-f3568a39f9e2 128 MiB Sat Apr 13 13:30:47 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:103 copied) 2024-04-13T13:43:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_37 info 2024-04-13T13:43:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_37 info' 2024-04-13T13:43:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_37 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_37': 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab852e6bfe 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab852e6bfe 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:30:59 2024 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:30:59 2024 2024-04-13T13:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:30:59 2024 2024-04-13T13:43:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: b1d1e9b9-23ac-4857-a6a7-e354f28ab5cd 2024-04-13T13:43:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_37 journal status 2024-04-13T13:43:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_37 journal status' 2024-04-13T13:43:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_37 2024-04-13T13:43:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_37 2024-04-13T13:43:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_37 snapshots 2024-04-13T13:43:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_37 snapshots' 2024-04-13T13:43:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_37 2024-04-13T13:43:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 110 .mirror.non_primary.b1d1e9b9-23ac-4857-a6a7-e354f28ab5cd.dc3e7239-df92-4735-b69a-0c0d0e7bacd6 128 MiB Sat Apr 13 13:31:00 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:104 copied) 2024-04-13T13:43:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_38 info 2024-04-13T13:43:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_38 info' 2024-04-13T13:43:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_38 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_38': 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e02a632ef4 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e02a632ef4 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:31:13 2024 2024-04-13T13:43:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:31:13 2024 2024-04-13T13:43:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:31:13 2024 2024-04-13T13:43:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: d96b2a28-af93-4496-b387-393648c236e0 2024-04-13T13:43:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_38 journal status 2024-04-13T13:43:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_38 journal status' 2024-04-13T13:43:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_38 2024-04-13T13:43:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_38 2024-04-13T13:43:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_38 snapshots 2024-04-13T13:43:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_38 snapshots' 2024-04-13T13:43:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_38 2024-04-13T13:43:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 111 .mirror.non_primary.d96b2a28-af93-4496-b387-393648c236e0.10fabde5-bbc0-489d-a2a0-aa12ffdce8e2 128 MiB Sat Apr 13 13:31:14 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:105 copied) 2024-04-13T13:43:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_39 info 2024-04-13T13:43:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_39 info' 2024-04-13T13:43:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_39 2024-04-13T13:43:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_39': 2024-04-13T13:43:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c7ef6f80a 2024-04-13T13:43:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c7ef6f80a 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:31:28 2024 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:31:28 2024 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:31:28 2024 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 41ea30c5-3df3-4041-8289-68ecbae98d63 2024-04-13T13:43:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_39 journal status 2024-04-13T13:43:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_39 journal status' 2024-04-13T13:43:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_39 2024-04-13T13:43:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_39 2024-04-13T13:43:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_39 snapshots' 2024-04-13T13:43:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_39 snapshots 2024-04-13T13:43:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_39 2024-04-13T13:43:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 112 .mirror.non_primary.41ea30c5-3df3-4041-8289-68ecbae98d63.22b51667-4beb-4a99-8b44-d3784d204d82 128 MiB Sat Apr 13 13:31:29 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:106 copied) 2024-04-13T13:43:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_4 info 2024-04-13T13:43:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_4 info' 2024-04-13T13:43:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_4 2024-04-13T13:43:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_4': 2024-04-13T13:43:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b3fdd556cc 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b3fdd556cc 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:34:56 2024 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:47 2024 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8d69fb4b-4707-45d2-bd35-f5dc2bafa56b 2024-04-13T13:43:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_4 journal status 2024-04-13T13:43:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_4 journal status' 2024-04-13T13:43:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_4 2024-04-13T13:43:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_4 2024-04-13T13:43:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_4 snapshots 2024-04-13T13:43:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_4 snapshots' 2024-04-13T13:43:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_4 2024-04-13T13:43:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 134 snap 128 MiB Sat Apr 13 13:34:52 2024 user 2024-04-13T13:43:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 135 .mirror.non_primary.8d69fb4b-4707-45d2-bd35-f5dc2bafa56b.03295f10-4151-402c-b3f0-c509a1168bae 128 MiB Sat Apr 13 13:34:53 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:125 copied) 2024-04-13T13:43:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_40 info 2024-04-13T13:43:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_40 info' 2024-04-13T13:43:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_40 2024-04-13T13:43:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_40': 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c23846c274 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c23846c274 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:31:41 2024 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:31:41 2024 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:31:41 2024 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8ed18fcc-ddf5-45b3-bceb-3cbe6226dcaa 2024-04-13T13:43:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_40 journal status 2024-04-13T13:43:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_40 journal status' 2024-04-13T13:43:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_40 2024-04-13T13:43:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_40 2024-04-13T13:43:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_40 snapshots 2024-04-13T13:43:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_40 snapshots' 2024-04-13T13:43:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_40 2024-04-13T13:43:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 113 .mirror.non_primary.8ed18fcc-ddf5-45b3-bceb-3cbe6226dcaa.caf093d9-70a3-4829-8300-8aec5841c133 128 MiB Sat Apr 13 13:31:42 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:107 copied) 2024-04-13T13:43:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_41 info' 2024-04-13T13:43:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_41 info 2024-04-13T13:43:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_41 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_41': 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab1b15ede9 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab1b15ede9 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:31:55 2024 2024-04-13T13:43:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:31:55 2024 2024-04-13T13:43:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:31:55 2024 2024-04-13T13:43:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 682df14d-9783-44dd-8781-b0f0d506d592 2024-04-13T13:43:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_41 journal status 2024-04-13T13:43:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_41 journal status' 2024-04-13T13:43:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_41 2024-04-13T13:43:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_41 2024-04-13T13:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_41 snapshots 2024-04-13T13:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_41 snapshots' 2024-04-13T13:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_41 2024-04-13T13:43:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 114 .mirror.non_primary.682df14d-9783-44dd-8781-b0f0d506d592.61a774ea-0391-488f-b763-e8546160e636 128 MiB Sat Apr 13 13:31:56 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:108 copied) 2024-04-13T13:43:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_42 info 2024-04-13T13:43:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_42 info' 2024-04-13T13:43:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_42 2024-04-13T13:43:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_42': 2024-04-13T13:43:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e08f29e0c3 2024-04-13T13:43:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e08f29e0c3 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:07 2024 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:07 2024 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:07 2024 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: a05e5ef4-930c-4ff6-b504-51ec542751fd 2024-04-13T13:43:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_42 journal status 2024-04-13T13:43:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_42 journal status' 2024-04-13T13:43:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_42 2024-04-13T13:43:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_42 2024-04-13T13:43:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_42 snapshots 2024-04-13T13:43:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_42 snapshots' 2024-04-13T13:43:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_42 2024-04-13T13:43:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 115 .mirror.non_primary.a05e5ef4-930c-4ff6-b504-51ec542751fd.564919b0-2ab0-4ab2-8c0c-f0a7efd424a1 128 MiB Sat Apr 13 13:32:07 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:109 copied) 2024-04-13T13:43:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_43 info 2024-04-13T13:43:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_43 info' 2024-04-13T13:43:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_43 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_43': 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c3c209b51 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c3c209b51 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:20 2024 2024-04-13T13:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:20 2024 2024-04-13T13:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:20 2024 2024-04-13T13:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: c0e8ec4e-880c-4448-9702-ba9518d9f196 2024-04-13T13:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_43 journal status 2024-04-13T13:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_43 journal status' 2024-04-13T13:43:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_43 2024-04-13T13:43:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_43 2024-04-13T13:43:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_43 snapshots 2024-04-13T13:43:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_43 snapshots' 2024-04-13T13:43:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_43 2024-04-13T13:43:34.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:34.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 116 .mirror.non_primary.c0e8ec4e-880c-4448-9702-ba9518d9f196.0057efce-a74d-4e71-a18b-a981d4b308ff 128 MiB Sat Apr 13 13:32:21 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:110 copied) 2024-04-13T13:43:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_44 info 2024-04-13T13:43:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_44 info' 2024-04-13T13:43:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_44 2024-04-13T13:43:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_44': 2024-04-13T13:43:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c2a9f2cd7f 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c2a9f2cd7f 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:33 2024 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:33 2024 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:33 2024 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 8f12f1ec-dc11-465e-a774-65eca20bf996 2024-04-13T13:43:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_44 journal status 2024-04-13T13:43:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_44 journal status' 2024-04-13T13:43:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_44 2024-04-13T13:43:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_44 2024-04-13T13:43:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_44 snapshots 2024-04-13T13:43:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_44 snapshots' 2024-04-13T13:43:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_44 2024-04-13T13:43:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 117 .mirror.non_primary.8f12f1ec-dc11-465e-a774-65eca20bf996.6066cf10-d8b3-4d3e-8fa8-e4079f4a8752 128 MiB Sat Apr 13 13:32:33 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:111 copied) 2024-04-13T13:43:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_45 info 2024-04-13T13:43:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_45 info' 2024-04-13T13:43:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_45 2024-04-13T13:43:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_45': 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab13564e23 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab13564e23 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:47 2024 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:47 2024 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:47 2024 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 4722198d-f141-466b-aec9-9f1bd91cfc3f 2024-04-13T13:43:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_45 journal status 2024-04-13T13:43:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_45 journal status' 2024-04-13T13:43:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_45 2024-04-13T13:43:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_45 2024-04-13T13:43:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_45 snapshots 2024-04-13T13:43:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_45 snapshots' 2024-04-13T13:43:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_45 2024-04-13T13:43:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 118 .mirror.non_primary.4722198d-f141-466b-aec9-9f1bd91cfc3f.650e4b05-454d-462a-8271-605c80d761c0 128 MiB Sat Apr 13 13:32:48 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:112 copied) 2024-04-13T13:43:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_46 info 2024-04-13T13:43:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_46 info' 2024-04-13T13:43:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_46 2024-04-13T13:43:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_46': 2024-04-13T13:43:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0d81d6e4b 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0d81d6e4b 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:32:53 2024 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:32:53 2024 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:32:53 2024 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: fe7f2378-9fe8-4c98-84bb-5958971e249f 2024-04-13T13:43:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_46 journal status 2024-04-13T13:43:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_46 journal status' 2024-04-13T13:43:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_46 2024-04-13T13:43:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_46 2024-04-13T13:43:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_46 snapshots 2024-04-13T13:43:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_46 snapshots' 2024-04-13T13:43:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_46 2024-04-13T13:43:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 119 .mirror.non_primary.fe7f2378-9fe8-4c98-84bb-5958971e249f.6532b67f-d03f-4cb1-b425-859cffa3c2d2 128 MiB Sat Apr 13 13:32:54 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:113 copied) 2024-04-13T13:43:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_47 info' 2024-04-13T13:43:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_47 info 2024-04-13T13:43:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_47 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_47': 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c7112e6e0 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c7112e6e0 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:33:06 2024 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:33:06 2024 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:33:06 2024 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1e77a3c2-50cb-4ed3-be13-ff3d35f16c46 2024-04-13T13:43:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_47 journal status 2024-04-13T13:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_47 journal status' 2024-04-13T13:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_47 2024-04-13T13:43:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_47 2024-04-13T13:43:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_47 snapshots 2024-04-13T13:43:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_47 snapshots' 2024-04-13T13:43:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_47 2024-04-13T13:43:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 120 .mirror.non_primary.1e77a3c2-50cb-4ed3-be13-ff3d35f16c46.254d2f44-502b-495f-86bf-c77d54485509 128 MiB Sat Apr 13 13:33:07 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:114 copied) 2024-04-13T13:43:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_48 info 2024-04-13T13:43:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_48 info' 2024-04-13T13:43:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_48 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_48': 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0c984d98e 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0c984d98e 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:33:42 2024 2024-04-13T13:43:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:33:42 2024 2024-04-13T13:43:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:33:42 2024 2024-04-13T13:43:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: fee9868f-6c82-4814-a3f0-2803aaff14e9 2024-04-13T13:43:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_48 journal status 2024-04-13T13:43:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_48 journal status' 2024-04-13T13:43:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_48 2024-04-13T13:43:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_48 2024-04-13T13:43:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_48 snapshots 2024-04-13T13:43:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_48 snapshots' 2024-04-13T13:43:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_48 2024-04-13T13:43:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 123 .mirror.non_primary.fee9868f-6c82-4814-a3f0-2803aaff14e9.6e68caab-2331-4a11-8be4-29fc34aebb2c 128 MiB Sat Apr 13 13:33:43 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:115 copied) 2024-04-13T13:43:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_49 info 2024-04-13T13:43:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_49 info' 2024-04-13T13:43:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_49 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_49': 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab6a4b011e 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab6a4b011e 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:33:28 2024 2024-04-13T13:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:33:28 2024 2024-04-13T13:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:33:28 2024 2024-04-13T13:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 89d28a1c-d65d-4fb9-b8cc-35062079a2b1 2024-04-13T13:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_49 journal status 2024-04-13T13:43:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_49 journal status' 2024-04-13T13:43:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_49 2024-04-13T13:43:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_49 2024-04-13T13:43:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_49 snapshots 2024-04-13T13:43:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_49 snapshots' 2024-04-13T13:43:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_49 2024-04-13T13:43:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 121 .mirror.non_primary.89d28a1c-d65d-4fb9-b8cc-35062079a2b1.0beec44b-698f-47df-a8c4-be648d451ffb 128 MiB Sat Apr 13 13:33:29 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:116 copied) 2024-04-13T13:43:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_5 info 2024-04-13T13:43:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_5 info' 2024-04-13T13:43:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_5 2024-04-13T13:43:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_5': 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab185c36c 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab185c36c 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:25:58 2024 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:35:04 2024 2024-04-13T13:43:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:25:58 2024 2024-04-13T13:43:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: b72ab703-5a35-4d41-90bc-db2c081ebc57 2024-04-13T13:43:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_5 journal status 2024-04-13T13:43:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_5 journal status' 2024-04-13T13:43:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_5 2024-04-13T13:43:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_5 2024-04-13T13:43:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_5 snapshots 2024-04-13T13:43:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_5 snapshots' 2024-04-13T13:43:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_5 2024-04-13T13:43:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 137 snap 128 MiB Sat Apr 13 13:35:00 2024 user 2024-04-13T13:43:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 138 .mirror.non_primary.b72ab703-5a35-4d41-90bc-db2c081ebc57.5f95bbfc-c1b3-4595-84be-72bc889056b8 128 MiB Sat Apr 13 13:35:01 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:127 copied) 2024-04-13T13:43:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_50 info 2024-04-13T13:43:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_50 info' 2024-04-13T13:43:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_50 2024-04-13T13:43:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_50': 2024-04-13T13:43:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2024-04-13T13:43:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0a4c3ac59 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0a4c3ac59 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:33:38 2024 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:33:38 2024 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:33:38 2024 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: bb8a902e-d932-412f-bc77-def5842a6663 2024-04-13T13:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_50 journal status 2024-04-13T13:43:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_50 journal status' 2024-04-13T13:43:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_50 2024-04-13T13:43:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_50 2024-04-13T13:43:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_50 snapshots 2024-04-13T13:43:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_50 snapshots' 2024-04-13T13:43:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_50 2024-04-13T13:43:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 122 .mirror.non_primary.bb8a902e-d932-412f-bc77-def5842a6663.10489371-3832-41d9-88dd-d61fe9287698 128 MiB Sat Apr 13 13:33:39 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:117 copied) 2024-04-13T13:43:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_6 info 2024-04-13T13:43:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_6 info' 2024-04-13T13:43:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_6 2024-04-13T13:43:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_6': 2024-04-13T13:43:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0d7251c0a 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0d7251c0a 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:13 2024 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:35:12 2024 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:13 2024 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 1466461b-b5d7-4bb3-9928-7a3a2fd5b22f 2024-04-13T13:43:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_6 journal status 2024-04-13T13:43:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_6 journal status' 2024-04-13T13:43:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_6 2024-04-13T13:43:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_6 2024-04-13T13:43:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_6 snapshots 2024-04-13T13:43:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_6 snapshots' 2024-04-13T13:43:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_6 2024-04-13T13:43:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 140 snap 128 MiB Sat Apr 13 13:35:08 2024 user 2024-04-13T13:43:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 141 .mirror.non_primary.1466461b-b5d7-4bb3-9928-7a3a2fd5b22f.0e19591e-8c82-436d-ac0f-20e5b8ff6249 128 MiB Sat Apr 13 13:35:09 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:129 copied) 2024-04-13T13:43:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_7 info 2024-04-13T13:43:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_7 info' 2024-04-13T13:43:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_7 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_7': 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 124c1410de1b 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.124c1410de1b 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:21 2024 2024-04-13T13:43:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:35:18 2024 2024-04-13T13:43:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:21 2024 2024-04-13T13:43:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 5e948815-4bc4-4300-914c-1855c84e9f33 2024-04-13T13:43:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_7 journal status 2024-04-13T13:43:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_7 journal status' 2024-04-13T13:43:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_7 2024-04-13T13:43:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_7 2024-04-13T13:43:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_7 snapshots 2024-04-13T13:43:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_7 snapshots' 2024-04-13T13:43:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_7 2024-04-13T13:43:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 143 snap 128 MiB Sat Apr 13 13:35:15 2024 user 2024-04-13T13:43:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 144 .mirror.non_primary.5e948815-4bc4-4300-914c-1855c84e9f33.323d6db4-91dd-4d62-9425-64b3ebd6194e 128 MiB Sat Apr 13 13:35:17 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:131 copied) 2024-04-13T13:43:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_8 info 2024-04-13T13:43:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_8 info' 2024-04-13T13:43:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_8 2024-04-13T13:43:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_8': 2024-04-13T13:43:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12b382b7609e 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12b382b7609e 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:34 2024 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:35:25 2024 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:34 2024 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 970b3a1e-6448-44f1-be2c-e3b2bc2b08af 2024-04-13T13:43:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_8 journal status 2024-04-13T13:43:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_8 journal status' 2024-04-13T13:43:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_8 2024-04-13T13:43:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_8 2024-04-13T13:43:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_8 snapshots 2024-04-13T13:43:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_8 snapshots' 2024-04-13T13:43:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_8 2024-04-13T13:43:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 146 snap 128 MiB Sat Apr 13 13:35:22 2024 user 2024-04-13T13:43:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 147 .mirror.non_primary.970b3a1e-6448-44f1-be2c-e3b2bc2b08af.9cafffc8-5294-4878-b501-fcfe31075e27 128 MiB Sat Apr 13 13:35:23 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:133 copied) 2024-04-13T13:43:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-04-13T13:43:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_9 info 2024-04-13T13:43:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_9 info' 2024-04-13T13:43:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_9 2024-04-13T13:43:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'image_9': 2024-04-13T13:43:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2024-04-13T13:43:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2024-04-13T13:43:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2024-04-13T13:43:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ab8dfa0264 2024-04-13T13:43:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ab8dfa0264 2024-04-13T13:43:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2024-04-13T13:43:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, non-primary 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sat Apr 13 13:26:41 2024 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sat Apr 13 13:35:32 2024 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sat Apr 13 13:26:41 2024 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: snapshot 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 9b0f1c92-41a8-4559-8a23-3818c789553a 2024-04-13T13:43:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2024-04-13T13:43:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_9 journal status 2024-04-13T13:43:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_9 journal status' 2024-04-13T13:43:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_9 2024-04-13T13:43:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image image_9 2024-04-13T13:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image image_9 snapshots 2024-04-13T13:43:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'image image_9 snapshots' 2024-04-13T13:43:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_9 2024-04-13T13:43:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-04-13T13:43:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 149 snap 128 MiB Sat Apr 13 13:35:30 2024 user 2024-04-13T13:43:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 150 .mirror.non_primary.9b0f1c92-41a8-4559-8a23-3818c789553a.3d4513f4-f226-47c4-a5a3-f22950c69f9f 128 MiB Sat Apr 13 13:35:31 2024 mirror (non-primary peer_uuids:[] 702f25c1-769a-41ec-a58d-5bd2b89bc57e:135 copied) 2024-04-13T13:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror rbd_mirroring omap vals 2024-04-13T13:43:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror rbd_mirroring omap vals' 2024-04-13T13:43:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2024-04-13T13:43:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_0c18f79e-f883-47a8-84ef-a7789e73b4ba 2024-04-13T13:43:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 65 64 33 32 35 66 39 64 |....124ced325f9d| 2024-04-13T13:43:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_1466461b-b5d7-4bb3-9928-7a3a2fd5b22f 2024-04-13T13:43:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 64 37 32 35 31 63 30 61 |....10e0d7251c0a| 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_173021c1-45fc-4ddf-ae55-e453164e52a8 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 65 64 62 37 61 35 34 66 |....124cedb7a54f| 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_1dd9564b-377b-4a17-82b5-1bee324ef4f2 2024-04-13T13:43:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 32 38 65 65 66 33 33 37 |....124c28eef337| 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_1e77a3c2-50cb-4ed3-be13-ff3d35f16c46 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 37 31 31 32 65 36 65 30 |....124c7112e6e0| 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_1f7e0537-71e2-4dcf-a741-47e1160880d2 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 35 36 66 39 39 65 35 37 |....10e056f99e57| 2024-04-13T13:43:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_213e24cb-89ef-475f-84d5-92800a3d1efb 2024-04-13T13:43:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 65 62 36 62 32 31 38 34 |....10abeb6b2184| 2024-04-13T13:43:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_37c03058-f2ca-48b7-b47d-ccfcb255694b 2024-04-13T13:43:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 63 32 39 30 32 65 37 37 37 64 |....12c2902e777d| 2024-04-13T13:43:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_41ea30c5-3df3-4041-8289-68ecbae98d63 2024-04-13T13:43:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 37 65 66 36 66 38 30 61 |....124c7ef6f80a| 2024-04-13T13:43:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_45670160-3031-494c-bc36-3e5d858b75c1 2024-04-13T13:43:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (14 bytes) : 2024-04-13T13:43:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0a 00 00 00 31 30 65 30 64 31 38 35 63 31 |....10e0d185c1| 2024-04-13T13:43:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000000e 2024-04-13T13:43:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_4722198d-f141-466b-aec9-9f1bd91cfc3f 2024-04-13T13:43:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 31 33 35 36 34 65 32 33 |....10ab13564e23| 2024-04-13T13:43:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_51d2605d-d07c-452b-98c9-76a5433b1837 2024-04-13T13:43:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (15 bytes) : 2024-04-13T13:43:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0b 00 00 00 31 30 61 62 32 64 31 64 39 38 31 |....10ab2d1d981| 2024-04-13T13:43:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000000f 2024-04-13T13:43:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_53de06e5-2f50-4532-ba7a-3aa9dd41bcfd 2024-04-13T13:43:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 62 33 66 30 64 37 32 37 38 39 |....12b3f0d72789| 2024-04-13T13:43:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_59ed4044-a6c7-40ba-a8ff-0113c71c89da 2024-04-13T13:43:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 32 31 66 65 37 37 63 37 |....124c21fe77c7| 2024-04-13T13:43:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_5b84301a-f350-4908-84da-ddb3ac371259 2024-04-13T13:43:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 62 33 62 64 66 32 65 30 38 33 |....12b3bdf2e083| 2024-04-13T13:43:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_5e948815-4bc4-4300-914c-1855c84e9f33 2024-04-13T13:43:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 31 34 31 30 64 65 31 62 |....124c1410de1b| 2024-04-13T13:43:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_682df14d-9783-44dd-8781-b0f0d506d592 2024-04-13T13:43:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 31 62 31 35 65 64 65 39 |....10ab1b15ede9| 2024-04-13T13:43:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_6c4ef21c-2a7c-465b-b23d-96050d4d8338 2024-04-13T13:43:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 62 33 38 62 63 65 64 35 37 63 |....12b38bced57c| 2024-04-13T13:43:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_7a6330ae-57ed-417b-a82d-4d987787e23b 2024-04-13T13:43:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 34 63 63 63 63 31 63 33 |....124c4cccc1c3| 2024-04-13T13:43:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_7cb77a9f-0f4a-4ba9-bc4b-ae0678719819 2024-04-13T13:43:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 65 33 34 64 34 61 30 65 |....10e0e34d4a0e| 2024-04-13T13:43:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_807d8287-a59d-4634-8f59-0370463ec872 2024-04-13T13:43:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 64 61 65 63 39 36 66 61 |....10e0daec96fa| 2024-04-13T13:43:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_89d28a1c-d65d-4fb9-b8cc-35062079a2b1 2024-04-13T13:43:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 36 61 34 62 30 31 31 65 |....10ab6a4b011e| 2024-04-13T13:43:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_8c509f7e-2bfc-4798-88c9-e8a036e7a227 2024-04-13T13:43:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (15 bytes) : 2024-04-13T13:43:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0b 00 00 00 31 32 34 63 31 61 32 64 33 37 61 |....124c1a2d37a| 2024-04-13T13:43:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000000f 2024-04-13T13:43:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_8d69fb4b-4707-45d2-bd35-f5dc2bafa56b 2024-04-13T13:43:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 62 33 66 64 64 35 35 36 63 63 |....12b3fdd556cc| 2024-04-13T13:43:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_8ed18fcc-ddf5-45b3-bceb-3cbe6226dcaa 2024-04-13T13:43:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 63 32 33 38 34 36 63 32 37 34 |....12c23846c274| 2024-04-13T13:43:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_8f12f1ec-dc11-465e-a774-65eca20bf996 2024-04-13T13:43:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 63 32 61 39 66 32 63 64 37 66 |....12c2a9f2cd7f| 2024-04-13T13:43:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_91c5558e-79d7-4410-a35b-b56664447d57 2024-04-13T13:43:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 62 33 63 65 32 34 30 33 64 66 |....12b3ce2403df| 2024-04-13T13:43:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_970b3a1e-6448-44f1-be2c-e3b2bc2b08af 2024-04-13T13:43:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 62 33 38 32 62 37 36 30 39 65 |....12b382b7609e| 2024-04-13T13:43:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_9b0f1c92-41a8-4559-8a23-3818c789553a 2024-04-13T13:43:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 38 64 66 61 30 32 36 34 |....10ab8dfa0264| 2024-04-13T13:43:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_9c3874e8-22df-4fdb-a27a-7e2740eed944 2024-04-13T13:43:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 65 39 38 39 39 62 32 35 |....10abe9899b25| 2024-04-13T13:43:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_a05e5ef4-930c-4ff6-b504-51ec542751fd 2024-04-13T13:43:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 38 66 32 39 65 30 63 33 |....10e08f29e0c3| 2024-04-13T13:43:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_b1d1e9b9-23ac-4857-a6a7-e354f28ab5cd 2024-04-13T13:43:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 38 35 32 65 36 62 66 65 |....10ab852e6bfe| 2024-04-13T13:43:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_b72ab703-5a35-4d41-90bc-db2c081ebc57 2024-04-13T13:43:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (15 bytes) : 2024-04-13T13:43:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0b 00 00 00 31 30 61 62 31 38 35 63 33 36 63 |....10ab185c36c| 2024-04-13T13:43:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000000f 2024-04-13T13:43:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_bb8a902e-d932-412f-bc77-def5842a6663 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 61 34 63 33 61 63 35 39 |....10e0a4c3ac59| 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 39 65 34 61 39 36 62 39 |....10ab9e4a96b9| 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_bf426828-2e69-4ba4-a59f-e3526fdc760d 2024-04-13T13:43:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 62 36 61 33 35 33 65 32 |....10abb6a353e2| 2024-04-13T13:43:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_c0e8ec4e-880c-4448-9702-ba9518d9f196 2024-04-13T13:43:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 33 63 32 30 39 62 35 31 |....124c3c209b51| 2024-04-13T13:43:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_c3ae73db-b27b-4d07-9b28-0323537a736a 2024-04-13T13:43:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 62 33 39 65 62 38 63 36 65 64 |....12b39eb8c6ed| 2024-04-13T13:43:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_c95975b5-0eee-4f76-8be8-10a90b792c79 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 63 32 63 65 66 65 64 64 33 62 |....12c2cefedd3b| 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_cc11b282-5f39-432a-9e34-fd7d24025087 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 32 37 30 65 33 38 32 33 |....10e0270e3823| 2024-04-13T13:43:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_d1a11424-da29-4237-abbf-b187574ee6bd 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 61 39 62 61 63 66 63 35 |....10aba9bacfc5| 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_d2b64f41-9de5-4eab-8f44-349912e93682 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 61 62 36 35 36 62 39 30 34 31 |....10ab656b9041| 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_d96b2a28-af93-4496-b387-393648c236e0 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 32 61 36 33 32 65 66 34 |....10e02a632ef4| 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_dc2077e0-ec0a-4427-ab08-fc9e2fa06c71 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 65 63 63 34 34 33 35 65 |....10e0ecc4435e| 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_df429b8a-ec65-47e3-b367-41f75bbe4da1 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 35 31 64 66 63 63 61 30 |....10e051dfcca0| 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_e58af553-340d-46ba-a87f-856f475da231 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 33 66 31 63 39 38 36 37 |....10e03f1c9867| 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_f38ae892-421e-4484-b8f6-de0cb55a7833 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 63 64 61 64 66 61 62 64 30 |....124cdadfabd0| 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_f666e830-e8d8-4600-a27f-f8027ab46f85 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 63 32 65 61 31 33 64 30 63 37 |....12c2ea13d0c7| 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_fe7f2378-9fe8-4c98-84bb-5958971e249f 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 64 38 31 64 36 65 34 62 |....10e0d81d6e4b| 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_fee9868f-6c82-4814-a3f0-2803aaff14e9 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 63 39 38 34 64 39 38 65 |....10e0c984d98e| 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab13564e23 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 37 32 32 31 39 |..*...$...472219| 2024-04-13T13:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 64 2d 66 31 34 31 2d 34 36 36 62 2d 61 65 63 |8d-f141-466b-aec| 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 39 66 31 62 64 39 31 63 66 63 33 66 01 01 |9-9f1bd91cfc3f..| 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab185c36c 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 37 32 61 62 37 |..*...$...b72ab7| 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 30 33 2d 35 61 33 35 2d 34 64 34 31 2d 39 30 62 |03-5a35-4d41-90b| 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 63 2d 64 62 32 63 30 38 31 65 62 63 35 37 01 01 |c-db2c081ebc57..| 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab1b15ede9 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 38 32 64 66 31 |..*...$...682df1| 2024-04-13T13:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 64 2d 39 37 38 33 2d 34 34 64 64 2d 38 37 38 |4d-9783-44dd-878| 2024-04-13T13:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 31 2d 62 30 66 30 64 35 30 36 64 35 39 32 01 01 |1-b0f0d506d592..| 2024-04-13T13:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab2d1d981 2024-04-13T13:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 31 64 32 36 30 |..*...$...51d260| 2024-04-13T13:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 35 64 2d 64 30 37 63 2d 34 35 32 62 2d 39 38 63 |5d-d07c-452b-98c| 2024-04-13T13:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 37 36 61 35 34 33 33 62 31 38 33 37 01 01 |9-76a5433b1837..| 2024-04-13T13:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab656b9041 2024-04-13T13:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 32 62 36 34 66 |..*...$...d2b64f| 2024-04-13T13:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 31 2d 39 64 65 35 2d 34 65 61 62 2d 38 66 34 |41-9de5-4eab-8f4| 2024-04-13T13:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 33 34 39 39 31 32 65 39 33 36 38 32 01 01 |4-349912e93682..| 2024-04-13T13:43:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab6a4b011e 2024-04-13T13:43:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 39 64 32 38 61 |..*...$...89d28a| 2024-04-13T13:43:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 63 2d 64 36 35 64 2d 34 66 62 39 2d 62 38 63 |1c-d65d-4fb9-b8c| 2024-04-13T13:43:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 63 2d 33 35 30 36 32 30 37 39 61 32 62 31 01 01 |c-35062079a2b1..| 2024-04-13T13:43:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab852e6bfe 2024-04-13T13:43:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 31 64 31 65 39 |..*...$...b1d1e9| 2024-04-13T13:43:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 39 2d 32 33 61 63 2d 34 38 35 37 2d 61 36 61 |b9-23ac-4857-a6a| 2024-04-13T13:43:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 37 2d 65 33 35 34 66 32 38 61 62 35 63 64 01 01 |7-e354f28ab5cd..| 2024-04-13T13:43:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab8dfa0264 2024-04-13T13:43:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 62 30 66 31 63 |..*...$...9b0f1c| 2024-04-13T13:43:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 39 32 2d 34 31 61 38 2d 34 35 35 39 2d 38 61 32 |92-41a8-4559-8a2| 2024-04-13T13:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 33 2d 33 38 31 38 63 37 38 39 35 35 33 61 01 01 |3-3818c789553a..| 2024-04-13T13:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ab9e4a96b9 2024-04-13T13:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 64 32 36 34 37 |..*...$...bd2647| 2024-04-13T13:43:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 65 2d 30 66 61 33 2d 34 39 63 33 2d 39 66 35 |be-0fa3-49c3-9f5| 2024-04-13T13:43:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 62 63 31 64 37 62 35 39 35 63 39 66 01 01 |f-bc1d7b595c9f..| 2024-04-13T13:43:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10aba9bacfc5 2024-04-13T13:43:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 31 61 31 31 34 |..*...$...d1a114| 2024-04-13T13:43:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 34 2d 64 61 32 39 2d 34 32 33 37 2d 61 62 62 |24-da29-4237-abb| 2024-04-13T13:43:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 62 31 38 37 35 37 34 65 65 36 62 64 01 01 |f-b187574ee6bd..| 2024-04-13T13:43:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10abb6a353e2 2024-04-13T13:43:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 66 34 32 36 38 |..*...$...bf4268| 2024-04-13T13:43:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 38 2d 32 65 36 39 2d 34 62 61 34 2d 61 35 39 |28-2e69-4ba4-a59| 2024-04-13T13:43:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 65 33 35 32 36 66 64 63 37 36 30 64 01 01 |f-e3526fdc760d..| 2024-04-13T13:43:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10abe9899b25 2024-04-13T13:43:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 63 33 38 37 34 |..*...$...9c3874| 2024-04-13T13:43:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 65 38 2d 32 32 64 66 2d 34 66 64 62 2d 61 32 37 |e8-22df-4fdb-a27| 2024-04-13T13:43:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 61 2d 37 65 32 37 34 30 65 65 64 39 34 34 01 01 |a-7e2740eed944..| 2024-04-13T13:43:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10abeb6b2184 2024-04-13T13:43:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 31 33 65 32 34 |..*...$...213e24| 2024-04-13T13:43:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 62 2d 38 39 65 66 2d 34 37 35 66 2d 38 34 64 |cb-89ef-475f-84d| 2024-04-13T13:43:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 35 2d 39 32 38 30 30 61 33 64 31 65 66 62 01 01 |5-92800a3d1efb..| 2024-04-13T13:43:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0270e3823 2024-04-13T13:43:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 63 31 31 62 32 |..*...$...cc11b2| 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 32 2d 35 66 33 39 2d 34 33 32 61 2d 39 65 33 |82-5f39-432a-9e3| 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 66 64 37 64 32 34 30 32 35 30 38 37 01 01 |4-fd7d24025087..| 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e02a632ef4 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 39 36 62 32 61 |..*...$...d96b2a| 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 38 2d 61 66 39 33 2d 34 34 39 36 2d 62 33 38 |28-af93-4496-b38| 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 37 2d 33 39 33 36 34 38 63 32 33 36 65 30 01 01 |7-393648c236e0..| 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e03f1c9867 2024-04-13T13:43:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 35 38 61 66 35 |..*...$...e58af5| 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 35 33 2d 33 34 30 64 2d 34 36 62 61 2d 61 38 37 |53-340d-46ba-a87| 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 38 35 36 66 34 37 35 64 61 32 33 31 01 01 |f-856f475da231..| 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e051dfcca0 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 66 34 32 39 62 |..*...$...df429b| 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 61 2d 65 63 36 35 2d 34 37 65 33 2d 62 33 36 |8a-ec65-47e3-b36| 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 37 2d 34 31 66 37 35 62 62 65 34 64 61 31 01 01 |7-41f75bbe4da1..| 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e056f99e57 2024-04-13T13:43:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 66 37 65 30 35 |..*...$...1f7e05| 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 33 37 2d 37 31 65 32 2d 34 64 63 66 2d 61 37 34 |37-71e2-4dcf-a74| 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 31 2d 34 37 65 31 31 36 30 38 38 30 64 32 01 01 |1-47e1160880d2..| 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e08f29e0c3 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 30 35 65 35 65 |..*...$...a05e5e| 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 34 2d 39 33 30 63 2d 34 66 66 36 2d 62 35 30 |f4-930c-4ff6-b50| 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 35 31 65 63 35 34 32 37 35 31 66 64 01 01 |4-51ec542751fd..| 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0a4c3ac59 2024-04-13T13:43:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 62 38 61 39 30 |..*...$...bb8a90| 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 65 2d 64 39 33 32 2d 34 31 32 66 2d 62 63 37 |2e-d932-412f-bc7| 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 37 2d 64 65 66 35 38 34 32 61 36 36 36 33 01 01 |7-def5842a6663..| 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0c984d98e 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 65 65 39 38 36 |..*...$...fee986| 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 66 2d 36 63 38 32 2d 34 38 31 34 2d 61 33 66 |8f-6c82-4814-a3f| 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 30 2d 32 38 30 33 61 61 66 66 31 34 65 39 01 01 |0-2803aaff14e9..| 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0d185c1 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 35 36 37 30 31 |..*...$...456701| 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 36 30 2d 33 30 33 31 2d 34 39 34 63 2d 62 63 33 |60-3031-494c-bc3| 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 36 2d 33 65 35 64 38 35 38 62 37 35 63 31 01 01 |6-3e5d858b75c1..| 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0d7251c0a 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 34 36 36 34 36 |..*...$...146646| 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 62 2d 62 35 64 37 2d 34 62 62 33 2d 39 39 32 |1b-b5d7-4bb3-992| 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 2d 37 61 33 61 32 66 64 35 62 32 32 66 01 01 |8-7a3a2fd5b22f..| 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0d81d6e4b 2024-04-13T13:43:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 65 37 66 32 33 |..*...$...fe7f23| 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 37 38 2d 39 66 65 38 2d 34 63 39 38 2d 38 34 62 |78-9fe8-4c98-84b| 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 62 2d 35 39 35 38 39 37 31 65 32 34 39 66 01 01 |b-5958971e249f..| 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0daec96fa 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 30 37 64 38 32 |..*...$...807d82| 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 37 2d 61 35 39 64 2d 34 36 33 34 2d 38 66 35 |87-a59d-4634-8f5| 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 30 33 37 30 34 36 33 65 63 38 37 32 01 01 |9-0370463ec872..| 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0e34d4a0e 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 63 62 37 37 61 |..*...$...7cb77a| 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 39 66 2d 30 66 34 61 2d 34 62 61 39 2d 62 63 34 |9f-0f4a-4ba9-bc4| 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 62 2d 61 65 30 36 37 38 37 31 39 38 31 39 01 01 |b-ae0678719819..| 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0ecc4435e 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 63 32 30 37 37 |..*...$...dc2077| 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 65 30 2d 65 63 30 61 2d 34 34 32 37 2d 61 62 30 |e0-ec0a-4427-ab0| 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 2d 66 63 39 65 32 66 61 30 36 63 37 31 01 01 |8-fc9e2fa06c71..| 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124c1410de1b 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 65 39 34 38 38 |..*...$...5e9488| 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 35 2d 34 62 63 34 2d 34 33 30 30 2d 39 31 34 |15-4bc4-4300-914| 2024-04-13T13:43:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 63 2d 31 38 35 35 63 38 34 65 39 66 33 33 01 01 |c-1855c84e9f33..| 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124c1a2d37a 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 63 35 30 39 66 |..*...$...8c509f| 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 37 65 2d 32 62 66 63 2d 34 37 39 38 2d 38 38 63 |7e-2bfc-4798-88c| 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 65 38 61 30 33 36 65 37 61 32 32 37 01 01 |9-e8a036e7a227..| 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124c21fe77c7 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 39 65 64 34 30 |..*...$...59ed40| 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 34 2d 61 36 63 37 2d 34 30 62 61 2d 61 38 66 |44-a6c7-40ba-a8f| 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 30 31 31 33 63 37 31 63 38 39 64 61 01 01 |f-0113c71c89da..| 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124c28eef337 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 64 64 39 35 36 |..*...$...1dd956| 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 62 2d 33 37 37 62 2d 34 61 31 37 2d 38 32 62 |4b-377b-4a17-82b| 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 35 2d 31 62 65 65 33 32 34 65 66 34 66 32 01 01 |5-1bee324ef4f2..| 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124c3c209b51 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 30 65 38 65 63 |..*...$...c0e8ec| 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 65 2d 38 38 30 63 2d 34 34 34 38 2d 39 37 30 |4e-880c-4448-970| 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 62 61 39 35 31 38 64 39 66 31 39 36 01 01 |2-ba9518d9f196..| 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124c4cccc1c3 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 61 36 33 33 30 |..*...$...7a6330| 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 61 65 2d 35 37 65 64 2d 34 31 37 62 2d 61 38 32 |ae-57ed-417b-a82| 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 2d 34 64 39 38 37 37 38 37 65 32 33 62 01 01 |d-4d987787e23b..| 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124c7112e6e0 2024-04-13T13:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 65 37 37 61 33 |..*...$...1e77a3| 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 32 2d 35 30 63 62 2d 34 65 64 33 2d 62 65 31 |c2-50cb-4ed3-be1| 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 33 2d 66 66 33 64 33 35 66 31 36 63 34 36 01 01 |3-ff3d35f16c46..| 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124c7ef6f80a 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 31 65 61 33 30 |..*...$...41ea30| 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 35 2d 33 64 66 33 2d 34 30 34 31 2d 38 32 38 |c5-3df3-4041-828| 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 36 38 65 63 62 61 65 39 38 64 36 33 01 01 |9-68ecbae98d63..| 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124cdadfabd0 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 33 38 61 65 38 |..*...$...f38ae8| 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 39 32 2d 34 32 31 65 2d 34 34 38 34 2d 62 38 66 |92-421e-4484-b8f| 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 36 2d 64 65 30 63 62 35 35 61 37 38 33 33 01 01 |6-de0cb55a7833..| 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124ced325f9d 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 63 31 38 66 37 |..*...$...0c18f7| 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 39 65 2d 66 38 38 33 2d 34 37 61 38 2d 38 34 65 |9e-f883-47a8-84e| 2024-04-13T13:43:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 61 37 37 38 39 65 37 33 62 34 62 61 01 01 |f-a7789e73b4ba..| 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_124cedb7a54f 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 37 33 30 32 31 |..*...$...173021| 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 31 2d 34 35 66 63 2d 34 64 64 66 2d 61 65 35 |c1-45fc-4ddf-ae5| 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 35 2d 65 34 35 33 31 36 34 65 35 32 61 38 01 01 |5-e453164e52a8..| 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12b382b7609e 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 37 30 62 33 61 |..*...$...970b3a| 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 65 2d 36 34 34 38 2d 34 34 66 31 2d 62 65 32 |1e-6448-44f1-be2| 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 63 2d 65 33 62 32 62 63 32 62 30 38 61 66 01 01 |c-e3b2bc2b08af..| 2024-04-13T13:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12b38bced57c 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 63 34 65 66 32 |..*...$...6c4ef2| 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 63 2d 32 61 37 63 2d 34 36 35 62 2d 62 32 33 |1c-2a7c-465b-b23| 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 2d 39 36 30 35 30 64 34 64 38 33 33 38 01 01 |d-96050d4d8338..| 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12b39eb8c6ed 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 33 61 65 37 33 |..*...$...c3ae73| 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 64 62 2d 62 32 37 62 2d 34 64 30 37 2d 39 62 32 |db-b27b-4d07-9b2| 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 2d 30 33 32 33 35 33 37 61 37 33 36 61 01 01 |8-0323537a736a..| 2024-04-13T13:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12b3bdf2e083 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 62 38 34 33 30 |..*...$...5b8430| 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 61 2d 66 33 35 30 2d 34 39 30 38 2d 38 34 64 |1a-f350-4908-84d| 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 61 2d 64 64 62 33 61 63 33 37 31 32 35 39 01 01 |a-ddb3ac371259..| 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12b3ce2403df 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 31 63 35 35 35 |..*...$...91c555| 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 65 2d 37 39 64 37 2d 34 34 31 30 2d 61 33 35 |8e-79d7-4410-a35| 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 62 2d 62 35 36 36 36 34 34 34 37 64 35 37 01 01 |b-b56664447d57..| 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12b3f0d72789 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 33 64 65 30 36 |..*...$...53de06| 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 65 35 2d 32 66 35 30 2d 34 35 33 32 2d 62 61 37 |e5-2f50-4532-ba7| 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 61 2d 33 61 61 39 64 64 34 31 62 63 66 64 01 01 |a-3aa9dd41bcfd..| 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12b3fdd556cc 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 64 36 39 66 62 |..*...$...8d69fb| 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 62 2d 34 37 30 37 2d 34 35 64 32 2d 62 64 33 |4b-4707-45d2-bd3| 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 35 2d 66 35 64 63 32 62 61 66 61 35 36 62 01 01 |5-f5dc2bafa56b..| 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12c23846c274 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 65 64 31 38 66 |..*...$...8ed18f| 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 63 2d 64 64 66 35 2d 34 35 62 33 2d 62 63 65 |cc-ddf5-45b3-bce| 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 62 2d 33 63 62 65 36 32 32 36 64 63 61 61 01 01 |b-3cbe6226dcaa..| 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12c2902e777d 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 37 63 30 33 30 |..*...$...37c030| 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 35 38 2d 66 32 63 61 2d 34 38 62 37 2d 62 34 37 |58-f2ca-48b7-b47| 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 2d 63 63 66 63 62 32 35 35 36 39 34 62 01 01 |d-ccfcb255694b..| 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12c2a9f2cd7f 2024-04-13T13:43:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 66 31 32 66 31 |..*...$...8f12f1| 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 65 63 2d 64 63 31 31 2d 34 36 35 65 2d 61 37 37 |ec-dc11-465e-a77| 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 36 35 65 63 61 32 30 62 66 39 39 36 01 01 |4-65eca20bf996..| 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12c2cefedd3b 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 39 35 39 37 35 |..*...$...c95975| 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 35 2d 30 65 65 65 2d 34 66 37 36 2d 38 62 65 |b5-0eee-4f76-8be| 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 2d 31 30 61 39 30 62 37 39 32 63 37 39 01 01 |8-10a90b792c79..| 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12c2ea13d0c7 2024-04-13T13:43:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 36 36 36 65 38 |..*...$...f666e8| 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 33 30 2d 65 38 64 38 2d 34 36 30 30 2d 61 32 37 |30-e8d8-4600-a27| 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 66 38 30 32 37 61 62 34 36 66 38 35 01 01 |f-f8027ab46f85..| 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_a2d8dee1-b6c0-44ee-9be5-600f2a9f2571 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2024-04-13T13:43:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 61 32 64 38 64 65 |..~...$...a2d8de| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 65 31 2d 62 36 63 30 2d 34 34 65 65 2d 39 62 65 |e1-b6c0-44ee-9be| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 35 2d 36 30 30 66 32 61 39 66 32 35 37 31 08 00 |5-600f2a9f2571..| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 37 30 32 66 32 35 63 31 |....$...702f25c1| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 37 36 39 61 2d 34 31 65 63 2d 61 35 38 64 2d |-769a-41ec-a58d-| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 35 62 64 32 62 38 39 62 63 35 37 65 08 8c 1a 66 |5bd2b89bc57e...f| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 7d 5f 76 0a |}_v.| 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2024-04-13T13:43:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 61 30 63 37 62 34 32 61 2d 37 62 37 32 2d 34 33 |a0c7b42a-7b72-43| 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 35 2d 61 62 37 39 2d 34 37 63 66 31 33 63 31 |25-ab79-47cf13c1| 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 61 62 63 62 |abcb| 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_0c18f79e-f883-47a8-84ef-a7789e73b4ba_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ab 0d f0 33 00 |...3.| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_1466461b-b5d7-4bb3-9928-7a3a2fd5b22f_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 10 61 36 00 00 |.a6..| 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_173021c1-45fc-4ddf-ae55-e453164e52a8_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 3d ab 7e 36 00 |=.~6.| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_1dd9564b-377b-4a17-82b5-1bee324ef4f2_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 e3 04 f2 33 00 |...3.| 2024-04-13T13:43:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_1e77a3c2-50cb-4ed3-be13-ff3d35f16c46_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 42 2f f3 33 00 |B/.3.| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_1f7e0537-71e2-4dcf-a741-47e1160880d2_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 d9 9f f4 33 00 |...3.| 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_213e24cb-89ef-475f-84d5-92800a3d1efb_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 af e5 f5 33 00 |...3.| 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_37c03058-f2ca-48b7-b47d-ccfcb255694b_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 5e 12 f7 33 00 |^..3.| 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_41ea30c5-3df3-4041-8289-68ecbae98d63_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 fc 14 63 31 00 |..c1.| 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_45670160-3031-494c-bc36-3e5d858b75c1_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 b9 78 f8 33 00 |.x.3.| 2024-04-13T13:43:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_4722198d-f141-466b-aec9-9f1bd91cfc3f_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 9b 0a 80 36 00 |...6.| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_51d2605d-d07c-452b-98c9-76a5433b1837_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 d8 9c f9 33 00 |...3.| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_53de06e5-2f50-4532-ba7a-3aa9dd41bcfd_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 d1 f0 37 00 00 |..7..| 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_59ed4044-a6c7-40ba-a8ff-0113c71c89da_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 6a 0a 81 36 00 |j..6.| 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_5b84301a-f350-4908-84da-ddb3ac371259_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 36 31 64 31 00 |61d1.| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_5e948815-4bc4-4300-914c-1855c84e9f33_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 07 b3 81 36 00 |...6.| 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_682df14d-9783-44dd-8781-b0f0d506d592_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 18 c0 fa 33 00 |...3.| 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_6c4ef21c-2a7c-465b-b23d-96050d4d8338_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 1b 14 39 00 00 |..9..| 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_7a6330ae-57ed-417b-a82d-4d987787e23b_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 9d 0d fc 33 00 |...3.| 2024-04-13T13:43:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_7cb77a9f-0f4a-4ba9-bc4b-ae0678719819_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 6d 2f 3a 00 00 |m/:..| 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_807d8287-a59d-4634-8f59-0370463ec872_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ac c3 64 31 00 |..d1.| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_89d28a1c-d65d-4fb9-b8cc-35062079a2b1_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 af 3f 3b 00 00 |.?;..| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_8c509f7e-2bfc-4798-88c9-e8a036e7a227_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 67 5e 82 36 00 |g^.6.| 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_8d69fb4b-4707-45d2-bd35-f5dc2bafa56b_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 97 3d 3c 00 00 |.=<..| 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_8ed18fcc-ddf5-45b3-bceb-3cbe6226dcaa_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 34 4b 65 31 00 |4Ke1.| 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_8f12f1ec-dc11-465e-a774-65eca20bf996_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 e9 08 83 36 00 |...6.| 2024-04-13T13:43:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_91c5558e-79d7-4410-a35b-b56664447d57_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 8a 67 3d 00 00 |.g=..| 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_970b3a1e-6448-44f1-be2c-e3b2bc2b08af_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 67 23 fd 33 00 |g#.3.| 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_9b0f1c92-41a8-4559-8a23-3818c789553a_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 e8 b5 83 36 00 |...6.| 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_9c3874e8-22df-4fdb-a27a-7e2740eed944_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 d5 d3 65 31 00 |..e1.| 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_a05e5ef4-930c-4ff6-b504-51ec542751fd_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 67 5f 66 31 00 |g_f1.| 2024-04-13T13:43:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_b1d1e9b9-23ac-4857-a6a7-e354f28ab5cd_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 3d cd 3e 00 00 |=.>..| 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_b72ab703-5a35-4d41-90bc-db2c081ebc57_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 a3 5d 84 36 00 |.].6.| 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_bb8a902e-d932-412f-bc77-def5842a6663_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 13 e8 66 31 00 |..f1.| 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 3e 14 00 00 00 00 00 00 01 |..,....>........| 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 9e 31 78 10 00 |............1x..| 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 00 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 bd 42 fe 33 00 |.B.3.| 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_bf426828-2e69-4ba4-a59f-e3526fdc760d_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 8a ff 84 36 00 |...6.| 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_c0e8ec4e-880c-4448-9702-ba9518d9f196_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 79 98 85 36 00 |y..6.| 2024-04-13T13:43:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_c3ae73db-b27b-4d07-9b28-0323537a736a_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 87 15 40 00 00 |..@..| 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_c95975b5-0eee-4f76-8be8-10a90b792c79_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 80 33 86 36 00 |.3.6.| 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_cc11b282-5f39-432a-9e34-fd7d24025087_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 b9 19 41 00 00 |..A..| 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_d1a11424-da29-4237-abbf-b187574ee6bd_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 0f 26 42 00 00 |.&B..| 2024-04-13T13:43:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_d2b64f41-9de5-4eab-8f44-349912e93682_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 fd 6f 43 00 00 |.oC..| 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_d96b2a28-af93-4496-b387-393648c236e0_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 1a 6a 67 31 00 |.jg1.| 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_dc2077e0-ec0a-4427-ab08-fc9e2fa06c71_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 67 11 00 00 00 00 00 00 01 |..,....g........| 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ac 46 8a 96 10 00 |...........F....| 2024-04-13T13:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 08 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 e3 9d 44 00 00 |..D..| 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_df429b8a-ec65-47e3-b367-41f75bbe4da1_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 a2 eb 67 31 00 |..g1.| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_e58af553-340d-46ba-a87f-856f475da231_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 a4 6f 68 31 00 |.oh1.| 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_f38ae892-421e-4484-b8f6-de0cb55a7833_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 be f1 68 31 00 |..h1.| 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_f666e830-e8d8-4600-a27f-f8027ab46f85_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ef 75 69 31 00 |.ui1.| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_fe7f2378-9fe8-4c98-84bb-5958971e249f_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 1b 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ee 18 66 7a 10 00 |............fz..| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 01 8c 1a 66 |rting replay...f| 2024-04-13T13:43:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 69 f5 69 31 00 |i.i1.| 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_fee9868f-6c82-4814-a3f0-2803aaff14e9_702f25c1-769a-41ec-a58d-5bd2b89bc57e 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (85 bytes) : 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 cd 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7d dc 80 b0 10 00 |..........}.....| 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f5 8b 1a 66 |rting replay...f| 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 0f d2 86 36 00 |...6.| 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000055 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_0c18f79e-f883-47a8-84ef-a7789e73b4ba 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 37 36 30 30 30 30 2e 30 |hot":133760000.0| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 37 36 30 30 30 30 |bytes":133760000| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 30 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |15405,"remote_sn| 2024-04-13T13:43:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 30 35 2c 22 72 65 |":1713015405,"re| 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d f6 8b 1a 66 ed 4e 0a 04 00 |e"}...f.N...| 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_1466461b-b5d7-4bb3-9928-7a3a2fd5b22f 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 33 30 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |15307,"remote_sn| 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 33 30 37 2c 22 72 65 |":1713015307,"re| 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d f6 8b 1a 66 b6 0a 0c 04 00 |e"}...f.....| 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_173021c1-45fc-4ddf-ae55-e453164e52a8 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 32 38 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15281,"remote_sn| 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 32 38 31 2c 22 72 65 |":1713015281,"re| 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d f6 8b 1a 66 73 47 0d 04 00 |e"}...fsG...| 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_1dd9564b-377b-4a17-82b5-1bee324ef4f2 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 30 33 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |15036,"remote_sn| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 30 33 36 2c 22 72 65 |":1713015036,"re| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d f6 8b 1a 66 39 68 0e 04 00 |e"}...f9h...| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_1e77a3c2-50cb-4ed3-be13-ff3d35f16c46 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 31 38 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |15184,"remote_sn| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 31 38 34 2c 22 72 65 |":1713015184,"re| 2024-04-13T13:43:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 e7 0e 98 39 00 |e"}...f...9.| 2024-04-13T13:43:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_1f7e0537-71e2-4dcf-a741-47e1160880d2 2024-04-13T13:43:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 30 33 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |15030,"remote_sn| 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 30 33 30 2c 22 72 65 |":1713015030,"re| 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d f6 8b 1a 66 9b d7 0f 04 00 |e"}...f.....| 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_213e24cb-89ef-475f-84d5-92800a3d1efb 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 38 30 30 39 36 30 2e 30 |hot":133800960.0| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 38 30 30 39 36 30 |bytes":133800960| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 32 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |15420,"remote_sn| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 32 30 2c 22 72 65 |":1713015420,"re| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 55 3e c5 38 00 |e"}...fU>.8.| 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_37c03058-f2ca-48b7-b47d-ccfcb255694b 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 30 34 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |15044,"remote_sn| 2024-04-13T13:43:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 30 34 34 2c 22 72 65 |":1713015044,"re| 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 b9 61 99 39 00 |e"}...f.a.9.| 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_41ea30c5-3df3-4041-8289-68ecbae98d63 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 30 38 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |15086,"remote_sn| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 30 38 36 2c 22 72 65 |":1713015086,"re| 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 78 d8 9a 39 00 |e"}...fx..9.| 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_45670160-3031-494c-bc36-3e5d858b75c1 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 32 37 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |15273,"remote_sn| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 32 37 33 2c 22 72 65 |":1713015273,"re| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d f6 8b 1a 66 49 0e 11 04 00 |e"}...fI....| 2024-04-13T13:43:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_4722198d-f141-466b-aec9-9f1bd91cfc3f 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 31 36 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |15165,"remote_sn| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 31 36 35 2c 22 72 65 |":1713015165,"re| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d fd 8b 1a 66 b7 48 c7 38 00 |e"}...f.H.8.| 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_51d2605d-d07c-452b-98c9-76a5433b1837 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 30 32 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15021,"remote_sn| 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 30 32 31 2c 22 72 65 |":1713015021,"re| 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d fd 8b 1a 66 94 89 c8 38 00 |e"}...f...8.| 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_53de06e5-2f50-4532-ba7a-3aa9dd41bcfd 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 39 32 39 39 38 34 2e 30 |hot":133929984.0| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 39 32 39 39 38 34 |bytes":133929984| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 33 38 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |15382,"remote_sn| 2024-04-13T13:43:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 33 38 32 2c 22 72 65 |":1713015382,"re| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d 07 8c 1a 66 33 71 fc 2e 00 |e"}...f3q...| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_59ed4044-a6c7-40ba-a8ff-0113c71c89da 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 33 34 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |15343,"remote_sn| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 33 34 33 2c 22 72 65 |":1713015343,"re| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 e8 4f 9c 39 00 |e"}...f.O.9.| 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_5b84301a-f350-4908-84da-ddb3ac371259 2024-04-13T13:43:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 38 35 36 32 35 36 2e 30 |hot":133856256.0| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 38 35 36 32 35 36 |bytes":133856256| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 33 35 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |15350,"remote_sn| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 33 35 30 2c 22 72 65 |":1713015350,"re| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 a5 ad c9 38 00 |e"}...f...8.| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_5e948815-4bc4-4300-914c-1855c84e9f33 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 33 31 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |15315,"remote_sn| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 33 31 35 2c 22 72 65 |":1713015315,"re| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d fd 8b 1a 66 0a e5 ca 38 00 |e"}...f...8.| 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_682df14d-9783-44dd-8781-b0f0d506d592 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 31 31 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |15113,"remote_sn| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 31 31 33 2c 22 72 65 |":1713015113,"re| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d fd 8b 1a 66 97 0c cc 38 00 |e"}...f...8.| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_6c4ef21c-2a7c-465b-b23d-96050d4d8338 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 32 35 30 36 36 32 34 2e 30 |hot":132506624.0| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 32 35 30 36 36 32 34 |bytes":132506624| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 37 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |15470,"remote_sn| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 37 30 2c 22 72 65 |":1713015470,"re| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d 07 8c 1a 66 61 4f fd 2e 00 |e"}...faO...| 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_7a6330ae-57ed-417b-a82d-4d987787e23b 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 33 36 37 38 30 38 2e 30 |hot":133367808.0| 2024-04-13T13:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 33 36 37 38 30 38 |bytes":133367808| 2024-04-13T13:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 36 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |15462,"remote_sn| 2024-04-13T13:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 36 32 2c 22 72 65 |":1713015462,"re| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 22 3a cd 38 00 |e"}...f":.8.| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_7cb77a9f-0f4a-4ba9-bc4b-ae0678719819 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 33 33 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |15336,"remote_sn| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 33 33 36 2c 22 72 65 |":1713015336,"re| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d f6 8b 1a 66 0c 32 12 04 00 |e"}...f.2...| 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_807d8287-a59d-4634-8f59-0370463ec872 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 32 34 30 38 33 32 30 2e 30 |hot":132408320.0| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 32 34 30 38 33 32 30 |bytes":132408320| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 33 39 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |15398,"remote_sn| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 33 39 38 2c 22 72 65 |":1713015398,"re| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 27 6b ce 38 00 |e"}...f'k.8.| 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_89d28a1c-d65d-4fb9-b8cc-35062079a2b1 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 32 30 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |15206,"remote_sn| 2024-04-13T13:43:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 32 30 36 2c 22 72 65 |":1713015206,"re| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 86 ea fd 2e 00 |e"}...f.....| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_8c509f7e-2bfc-4798-88c9-e8a036e7a227 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 34 31 38 34 39 36 30 2e 30 |hot":134184960.0| 2024-04-13T13:43:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 38 34 39 36 30 |bytes":134184960| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 33 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |15434,"remote_sn| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 33 34 2c 22 72 65 |":1713015434,"re| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d f6 8b 1a 66 3e 50 13 04 00 |e"}...f>P...| 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_8d69fb4b-4707-45d2-bd35-f5dc2bafa56b 2024-04-13T13:43:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 32 39 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15291,"remote_sn| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 32 39 31 2c 22 72 65 |":1713015291,"re| 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 44 42 9d 39 00 |e"}...fDB.9.| 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_8ed18fcc-ddf5-45b3-bceb-3cbe6226dcaa 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 30 39 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |15099,"remote_sn| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 30 39 39 2c 22 72 65 |":1713015099,"re| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 86 7b fe 2e 00 |e"}...f.{...| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_8f12f1ec-dc11-465e-a774-65eca20bf996 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 31 35 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15151,"remote_sn| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 31 35 31 2c 22 72 65 |":1713015151,"re| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 89 29 9e 39 00 |e"}...f.).9.| 2024-04-13T13:43:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_91c5558e-79d7-4410-a35b-b56664447d57 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 34 30 32 36 32 34 2e 30 |hot":133402624.0| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 34 30 32 36 32 34 |bytes":133402624| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 34 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15441,"remote_sn| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 34 31 2c 22 72 65 |":1713015441,"re| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d 07 8c 1a 66 8d 4c ff 2e 00 |e"}...f.L...| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_970b3a1e-6448-44f1-be2c-e3b2bc2b08af 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 33 32 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |15322,"remote_sn| 2024-04-13T13:43:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 33 32 32 2c 22 72 65 |":1713015322,"re| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d f6 8b 1a 66 48 72 14 04 00 |e"}...fHr...| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_9b0f1c92-41a8-4559-8a23-3818c789553a 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 33 32 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |15329,"remote_sn| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 33 32 39 2c 22 72 65 |":1713015329,"re| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 3c da 9e 39 00 |e"}...f<..9.| 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_9c3874e8-22df-4fdb-a27a-7e2740eed944 2024-04-13T13:43:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 31 36 35 32 36 30 38 2e 30 |hot":131652608.0| 2024-04-13T13:43:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 31 36 35 32 36 30 38 |bytes":131652608| 2024-04-13T13:43:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 34 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |15448,"remote_sn| 2024-04-13T13:43:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 34 38 2c 22 72 65 |":1713015448,"re| 2024-04-13T13:43:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 29 bf cf 38 00 |e"}...f)..8.| 2024-04-13T13:43:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_a05e5ef4-930c-4ff6-b504-51ec542751fd 2024-04-13T13:43:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 31 32 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |15125,"remote_sn| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 31 32 35 2c 22 72 65 |":1713015125,"re| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 dc d6 ff 2e 00 |e"}...f.....| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_b1d1e9b9-23ac-4857-a6a7-e354f28ab5cd 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 30 35 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |15057,"remote_sn| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 30 35 37 2c 22 72 65 |":1713015057,"re| 2024-04-13T13:43:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 0d 67 00 2f 00 |e"}...f.g./.| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_b72ab703-5a35-4d41-90bc-db2c081ebc57 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 32 39 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |15299,"remote_sn| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 32 39 39 2c 22 72 65 |":1713015299,"re| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d f6 8b 1a 66 45 a0 15 04 00 |e"}...fE....| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_bb8a902e-d932-412f-bc77-def5842a6663 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 32 31 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |15217,"remote_sn| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 32 31 37 2c 22 72 65 |":1713015217,"re| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 e1 0a 01 2f 00 |e"}...f.../.| 2024-04-13T13:43:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 32 36 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |15265,"remote_sn| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 32 36 35 2c 22 72 65 |":1713015265,"re| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 5a 9b 01 2f 00 |e"}...fZ../.| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_bf426828-2e69-4ba4-a59f-e3526fdc760d 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 33 35 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |15357,"remote_sn| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 33 35 37 2c 22 72 65 |":1713015357,"re| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 73 81 9f 39 00 |e"}...fs..9.| 2024-04-13T13:43:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_c0e8ec4e-880c-4448-9702-ba9518d9f196 2024-04-13T13:43:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 31 33 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |15138,"remote_sn| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 31 33 38 2c 22 72 65 |":1713015138,"re| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 1e bb a0 39 00 |e"}...f...9.| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_c3ae73db-b27b-4d07-9b28-0323537a736a 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 34 31 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |15413,"remote_sn| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 34 31 33 2c 22 72 65 |":1713015413,"re| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 31 2c a2 39 00 |e"}...f1,.9.| 2024-04-13T13:43:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_c95975b5-0eee-4f76-8be8-10a90b792c79 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 38 37 32 36 34 30 2e 30 |hot":133872640.0| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 38 37 32 36 34 30 |bytes":133872640| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 35 30 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15501,"remote_sn| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 35 30 31 2c 22 72 65 |":1713015501,"re| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d 00 8c 1a 66 46 98 a3 39 00 |e"}...fF..9.| 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_cc11b282-5f39-432a-9e34-fd7d24025087 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 31 39 38 36 34 33 32 2e 30 |hot":131986432.0| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 31 39 38 36 34 33 32 |bytes":131986432| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 35 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |15455,"remote_sn| 2024-04-13T13:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 35 35 2c 22 72 65 |":1713015455,"re| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 b1 e1 d0 38 00 |e"}...f...8.| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_d1a11424-da29-4237-abbf-b187574ee6bd 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 34 30 30 35 37 36 2e 30 |hot":133400576.0| 2024-04-13T13:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 34 30 30 35 37 36 |bytes":133400576| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 37 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |15478,"remote_sn| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 37 38 2c 22 72 65 |":1713015478,"re| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 eb 04 d2 38 00 |e"}...f...8.| 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_d2b64f41-9de5-4eab-8f44-349912e93682 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 37 31 35 39 36 38 2e 30 |hot":133715968.0| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 37 31 35 39 36 38 |bytes":133715968| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 33 39 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |15390,"remote_sn| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 33 39 30 2c 22 72 65 |":1713015390,"re| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d 07 8c 1a 66 aa 23 02 2f 00 |e"}...f.#./.| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_d96b2a28-af93-4496-b387-393648c236e0 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 48 14 00 00 00 00 00 00 01 |..,....H........| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 fc 6b e7 10 00 |............k...| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 30 37 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15071,"remote_sn| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 30 37 31 2c 22 72 65 |":1713015071,"re| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 00 8c 1a 66 a7 27 a5 39 00 |e"}...f.'.9.| 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_dc2077e0-ec0a-4427-ab08-fc9e2fa06c71 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 34 30 30 33 37 31 32 2e 30 |hot":134003712.0| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 30 33 37 31 32 |bytes":134003712| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 33 36 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |15364,"remote_sn| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 33 36 34 2c 22 72 65 |":1713015364,"re| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 cc 28 d3 38 00 |e"}...f.(.8.| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_df429b8a-ec65-47e3-b367-41f75bbe4da1 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 35 33 39 38 34 30 2e 30 |hot":133539840.0| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 35 33 39 38 34 30 |bytes":133539840| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 38 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |15486,"remote_sn| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 38 36 2c 22 72 65 |":1713015486,"re| 2024-04-13T13:43:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d f6 8b 1a 66 80 be 16 04 00 |e"}...f.....| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_e58af553-340d-46ba-a87f-856f475da231 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ac 13 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 eb 89 3b 10 00 |.............;..| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 33 35 33 37 37 39 32 2e 30 |hot":133537792.0| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 35 33 37 37 39 32 |bytes":133537792| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 32 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |15427,"remote_sn| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 32 37 2c 22 72 65 |":1713015427,"re| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d f6 8b 1a 66 9e e6 17 04 00 |e"}...f.....| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_f38ae892-421e-4484-b8f6-de0cb55a7833 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 33 37 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15371,"remote_sn| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 33 37 31 2c 22 72 65 |":1713015371,"re| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 bc ab 02 2f 00 |e"}...f.../.| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_f666e830-e8d8-4600-a27f-f8027ab46f85 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (300 bytes) : 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 12 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 84 be 7e 7e 10 00 |............~~..| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 31 33 34 31 30 32 30 31 36 2e 30 |hot":134102016.0| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 30 32 30 31 36 |bytes":134102016| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 31 35 34 39 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |15494,"remote_sn| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 22 3a 31 37 31 33 30 31 35 34 39 34 2c 22 72 65 |":1713015494,"re| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 65 22 7d fd 8b 1a 66 5a 52 d4 38 00 |e"}...fZR.8.| 2024-04-13T13:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000012c 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_fe7f2378-9fe8-4c98-84bb-5958971e249f 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 31 37 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |15171,"remote_sn| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 31 37 31 2c 22 72 65 |":1713015171,"re| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 b1 2f 03 2f 00 |e"}...f././.| 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_fee9868f-6c82-4814-a3f0-2803aaff14e9 2024-04-13T13:43:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (284 bytes) : 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 e0 10 00 00 00 00 00 00 01 |..,.............| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e6 5f ff 39 10 00 |..........._.9..| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 33 30 |timestamp":17130| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 35 31 39 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |15192,"remote_sn| 2024-04-13T13:43:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-04-13T13:43:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 22 3a 31 37 31 33 30 31 35 31 39 32 2c 22 72 65 |":1713015192,"re| 2024-04-13T13:43:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-04-13T13:43:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 65 22 7d 07 8c 1a 66 da b7 03 2f 00 |e"}...f.../.| 2024-04-13T13:43:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000011c 2024-04-13T13:43:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2024-04-13T13:43:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror rbd_mirror_leader omap vals 2024-04-13T13:43:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2024-04-13T13:43:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_0c18f79e-f883-47a8-84ef-a7789e73b4ba 2024-04-13T13:43:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 52 8a |..........5036R.| 2024-04-13T13:43:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 04 61 bd 38 0a 00 00 00 01 01 04 00 00 00 |.f.a.8..........| 2024-04-13T13:43:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_1466461b-b5d7-4bb3-9928-7a3a2fd5b22f 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 52 8a |..........5036R.| 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 97 74 bd 38 0a 00 00 00 01 01 04 00 00 00 |.f.t.8..........| 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_173021c1-45fc-4ddf-ae55-e453164e52a8 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 52 8a |..........5036R.| 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 82 7f bd 38 0a 00 00 00 01 01 04 00 00 00 |.f...8..........| 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_1dd9564b-377b-4a17-82b5-1bee324ef4f2 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 39 8a |..........50369.| 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 82 29 e9 36 0a 00 00 00 01 01 04 00 00 00 |.f.).6..........| 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_1e77a3c2-50cb-4ed3-be13-ff3d35f16c46 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 74 91 0c 3a 0a 00 00 00 01 01 04 00 00 00 |.ft..:..........| 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_1f7e0537-71e2-4dcf-a741-47e1160880d2 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 39 8a |..........50369.| 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 c8 3d e9 36 0a 00 00 00 01 01 04 00 00 00 |.f.=.6..........| 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_213e24cb-89ef-475f-84d5-92800a3d1efb 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 fb 89 |..........4846..| 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 21 b2 ac 38 0a 00 00 00 01 01 04 00 00 00 |.f!..8..........| 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_37c03058-f2ca-48b7-b47d-ccfcb255694b 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 39 bf 0c 3a 0a 00 00 00 01 01 04 00 00 00 |.f9..:..........| 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_41ea30c5-3df3-4041-8289-68ecbae98d63 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 95 8a |..........5192..| 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 9a 16 60 3b 0a 00 00 00 01 01 04 00 00 00 |.f..`;..........| 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_45670160-3031-494c-bc36-3e5d858b75c1 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 52 8a |..........5036R.| 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 db 8c bd 38 0a 00 00 00 01 01 04 00 00 00 |.f...8..........| 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_4722198d-f141-466b-aec9-9f1bd91cfc3f 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 fb 89 |..........4846..| 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 ec ce ac 38 0a 00 00 00 01 01 04 00 00 00 |.f...8..........| 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_51d2605d-d07c-452b-98c9-76a5433b1837 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 fb 89 |..........4846..| 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 6c dd ac 38 0a 00 00 00 01 01 04 00 00 00 |.fl..8..........| 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_53de06e5-2f50-4532-ba7a-3aa9dd41bcfd 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 52 8a |..........4320R.| 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 b2 af bd 38 0a 00 00 00 01 01 04 00 00 00 |.f...8..........| 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_59ed4044-a6c7-40ba-a8ff-0113c71c89da 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 51 f7 0c 3a 0a 00 00 00 01 01 04 00 00 00 |.fQ..:..........| 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_5b84301a-f350-4908-84da-ddb3ac371259 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 fb 89 |..........4846..| 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 ca ed ac 38 0a 00 00 00 01 01 04 00 00 00 |.f...8..........| 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_5e948815-4bc4-4300-914c-1855c84e9f33 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 e7 89 |..........4846..| 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 93 7e 1e 37 0a 00 00 00 01 01 04 00 00 00 |.f.~.7..........| 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_682df14d-9783-44dd-8781-b0f0d506d592 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 fb 89 |..........4846..| 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 44 fe ac 38 0a 00 00 00 01 01 04 00 00 00 |.fD..8..........| 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_6c4ef21c-2a7c-465b-b23d-96050d4d8338 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 b4 89 |..........4320..| 2024-04-13T13:43:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 e7 67 a1 30 0a 00 00 00 01 01 04 00 00 00 |.f.g.0..........| 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_7a6330ae-57ed-417b-a82d-4d987787e23b 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 e7 89 |..........4846..| 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 f6 91 1e 37 0a 00 00 00 01 01 04 00 00 00 |.f...7..........| 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_7cb77a9f-0f4a-4ba9-bc4b-ae0678719819 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 52 8a |..........5036R.| 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 0a c6 bd 38 0a 00 00 00 01 01 04 00 00 00 |.f...8..........| 2024-04-13T13:43:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_807d8287-a59d-4634-8f59-0370463ec872 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 e7 89 |..........4846..| 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 94 9d 1e 37 0a 00 00 00 01 01 04 00 00 00 |.f...7..........| 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_89d28a1c-d65d-4fb9-b8cc-35062079a2b1 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 fb 89 |..........4320..| 2024-04-13T13:43:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 09 10 ad 38 0a 00 00 00 01 01 04 00 00 00 |.f...8..........| 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_8c509f7e-2bfc-4798-88c9-e8a036e7a227 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 39 8a |..........50369.| 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 43 71 e9 36 0a 00 00 00 01 01 04 00 00 00 |.fCq.6..........| 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_8d69fb4b-4707-45d2-bd35-f5dc2bafa56b 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 53 27 0d 3a 0a 00 00 00 01 01 04 00 00 00 |.fS'.:..........| 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_8ed18fcc-ddf5-45b3-bceb-3cbe6226dcaa 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 52 8a |..........4320R.| 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 48 d2 bd 38 0a 00 00 00 01 01 04 00 00 00 |.fH..8..........| 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_8f12f1ec-dc11-465e-a774-65eca20bf996 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 ca 40 0d 3a 0a 00 00 00 01 01 04 00 00 00 |.f.@.:..........| 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_91c5558e-79d7-4410-a35b-b56664447d57 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 b4 89 |..........4320..| 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 dc 9c a1 30 0a 00 00 00 01 01 04 00 00 00 |.f...0..........| 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_970b3a1e-6448-44f1-be2c-e3b2bc2b08af 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 39 8a |..........50369.| 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 2f cb e9 36 0a 00 00 00 01 01 04 00 00 00 |.f/..6..........| 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_9b0f1c92-41a8-4559-8a23-3818c789553a 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 ab b2 0d 3a 0a 00 00 00 01 01 04 00 00 00 |.f...:..........| 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_9c3874e8-22df-4fdb-a27a-7e2740eed944 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 fb 89 |..........4846..| 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 4a 31 ad 38 0a 00 00 00 01 01 04 00 00 00 |.fJ1.8..........| 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_a05e5ef4-930c-4ff6-b504-51ec542751fd 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 56 89 |..........4320V.| 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 03 34 5c 03 0a 00 00 00 01 01 04 00 00 00 |.f.4\...........| 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_b1d1e9b9-23ac-4857-a6a7-e354f28ab5cd 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 fb 89 |..........4320..| 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 ff 3e ad 38 0a 00 00 00 01 01 04 00 00 00 |.f.>.8..........| 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_b72ab703-5a35-4d41-90bc-db2c081ebc57 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 52 8a |..........5036R.| 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 61 eb bd 38 0a 00 00 00 01 01 04 00 00 00 |.fa..8..........| 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_bb8a902e-d932-412f-bc77-def5842a6663 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 b1 89 |..........4320..| 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 5c 64 7d 30 0a 00 00 00 01 01 04 00 00 00 |.f\d}0..........| 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_bd2647be-0fa3-49c3-9f5f-bc1d7b595c9f 2024-04-13T13:43:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 52 8a |..........4320R.| 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 7e f5 bd 38 0a 00 00 00 01 01 04 00 00 00 |.f~..8..........| 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_bf426828-2e69-4ba4-a59f-e3526fdc760d 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 c8 d3 0d 3a 0a 00 00 00 01 01 04 00 00 00 |.f...:..........| 2024-04-13T13:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_c0e8ec4e-880c-4448-9702-ba9518d9f196 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 91 ea 0d 3a 0a 00 00 00 01 01 04 00 00 00 |.f...:..........| 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_c3ae73db-b27b-4d07-9b28-0323537a736a 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 4f 00 0e 3a 0a 00 00 00 01 01 04 00 00 00 |.fO..:..........| 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_c95975b5-0eee-4f76-8be8-10a90b792c79 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 92 8a |..........5192..| 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 cd 14 0e 3a 0a 00 00 00 01 01 04 00 00 00 |.f...:..........| 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_cc11b282-5f39-432a-9e34-fd7d24025087 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 e7 89 |..........4846..| 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 03 cf 1e 37 0a 00 00 00 01 01 04 00 00 00 |.f...7..........| 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_d1a11424-da29-4237-abbf-b187574ee6bd 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 fb 89 |..........4846..| 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 c1 5f ad 38 0a 00 00 00 01 01 04 00 00 00 |.f._.8..........| 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_d2b64f41-9de5-4eab-8f44-349912e93682 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 fb 89 |..........4320..| 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 29 6e ad 38 0a 00 00 00 01 01 04 00 00 00 |.f)n.8..........| 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_d96b2a28-af93-4496-b387-393648c236e0 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 39 32 7f 8a |..........5192..| 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 ef a5 e9 38 0a 00 00 00 01 01 04 00 00 00 |.f...8..........| 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_dc2077e0-ec0a-4427-ab08-fc9e2fa06c71 2024-04-13T13:43:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 e7 89 |..........4846..| 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 c6 e7 1e 37 0a 00 00 00 01 01 04 00 00 00 |.f...7..........| 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_df429b8a-ec65-47e3-b367-41f75bbe4da1 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 39 8a |..........50369.| 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 bc 01 ea 36 0a 00 00 00 01 01 04 00 00 00 |.f...6..........| 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_e58af553-340d-46ba-a87f-856f475da231 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 30 33 36 39 8a |..........50369.| 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 18 12 ea 36 0a 00 00 00 01 01 04 00 00 00 |.f...6..........| 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_f38ae892-421e-4484-b8f6-de0cb55a7833 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 95 8a |..........4320..| 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 2c 49 60 3b 0a 00 00 00 01 01 04 00 00 00 |.f,I`;..........| 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_f666e830-e8d8-4600-a27f-f8027ab46f85 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 34 36 92 8a |..........4846..| 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 13 46 0e 3a 0a 00 00 00 01 01 04 00 00 00 |.f.F.:..........| 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_fe7f2378-9fe8-4c98-84bb-5958971e249f 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 84 89 |..........4320..| 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 ed ab 68 27 0a 00 00 00 01 01 04 00 00 00 |.f..h'..........| 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_map_fee9868f-6c82-4814-a3f0-2803aaff14e9 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 32 30 b5 89 |..........4320..| 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 1a 66 27 ea bd 30 0a 00 00 00 01 01 04 00 00 00 |.f'..0..........| 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 00 00 |....| 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_4320 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_4846 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_5036 2024-04-13T13:43:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_5192 2024-04-13T13:43:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T13:43:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns1 images 2024-04-13T13:43:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror ns1 images' 2024-04-13T13:43:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2024-04-13T13:43:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns1 mirror pool info 2024-04-13T13:43:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirrorns1 mirror pool info' 2024-04-13T13:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2024-04-13T13:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2024-04-13T13:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns1 mirror pool status 2024-04-13T13:43:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirrorns1 mirror pool status' 2024-04-13T13:43:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: OK 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 0 total 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4317: 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4320 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4843: 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4846 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 5033: 2024-04-13T13:43:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 5036 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 5189: 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 5192 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2024-04-13T13:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2024-04-13T13:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2024-04-13T13:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror ns1 rbd_mirroring omap vals' 2024-04-13T13:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2024-04-13T13:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 65 30 64 39 39 33 61 62 2d 65 37 30 66 2d 34 33 |e0d993ab-e70f-43| 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 39 33 2d 38 34 65 66 2d 36 35 39 30 35 35 38 61 |93-84ef-6590558a| 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 34 65 39 |84e9| 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror ns1 rbd_mirror_leader omap vals' 2024-04-13T13:43:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2024-04-13T13:43:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2024-04-13T13:43:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_4320 2024-04-13T13:43:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_4846 2024-04-13T13:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_5192 2024-04-13T13:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T13:43:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns2 images 2024-04-13T13:43:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror ns2 images' 2024-04-13T13:43:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2024-04-13T13:43:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns2 mirror pool info 2024-04-13T13:43:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirrorns2 mirror pool info' 2024-04-13T13:43:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2024-04-13T13:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2024-04-13T13:43:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns2 mirror pool status 2024-04-13T13:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirrorns2 mirror pool status' 2024-04-13T13:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2024-04-13T13:43:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: OK 2024-04-13T13:43:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2024-04-13T13:43:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2024-04-13T13:43:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 0 total 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4317: 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4320 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4843: 2024-04-13T13:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4846 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 5033: 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 5036 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 5189: 2024-04-13T13:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 5192 2024-04-13T13:43:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2024-04-13T13:43:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2024-04-13T13:43:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2024-04-13T13:43:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2024-04-13T13:43:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror ns2 rbd_mirroring omap vals' 2024-04-13T13:43:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 63 38 30 36 65 30 32 64 2d 61 32 63 62 2d 34 63 |c806e02d-a2cb-4c| 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 65 39 2d 62 64 32 36 2d 65 34 30 33 33 38 61 30 |e9-bd26-e40338a0| 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 30 36 31 36 |0616| 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2024-04-13T13:43:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror ns2 rbd_mirror_leader omap vals' 2024-04-13T13:43:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2024-04-13T13:43:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_4320 2024-04-13T13:43:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_4846 2024-04-13T13:43:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_5192 2024-04-13T13:43:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-04-13T13:43:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T13:43:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent images 2024-04-13T13:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent images' 2024-04-13T13:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' ls -l 2024-04-13T13:43:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent mirror pool info 2024-04-13T13:43:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent mirror pool info' 2024-04-13T13:43:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool info 2024-04-13T13:43:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2024-04-13T13:43:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster1 2024-04-13T13:43:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2024-04-13T13:43:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: 963bf1d1-056a-441a-b7b6-480b4924a6c2 2024-04-13T13:43:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster2 2024-04-13T13:43:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: 6c9f034a-5e1c-492a-acc5-df0cc2152dd3 2024-04-13T13:43:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2024-04-13T13:43:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2024-04-13T13:43:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent mirror pool status 2024-04-13T13:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent mirror pool status' 2024-04-13T13:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool status --verbose 2024-04-13T13:43:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: OK 2024-04-13T13:43:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 0 total 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4317: 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4322 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4843: 2024-04-13T13:43:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4849 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 5033: 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 5039 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 5189: 2024-04-13T13:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 5194 2024-04-13T13:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2024-04-13T13:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2024-04-13T13:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 18.2.2-1187-gd540ebac 2024-04-13T13:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2024-04-13T13:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2024-04-13T13:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2024-04-13T13:43:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls 2024-04-13T13:43:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2024-04-13T13:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent rbd_mirroring omap vals' 2024-04-13T13:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_963bf1d1-056a-441a-b7b6-480b4924a6c2 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 39 36 33 62 66 31 |..~...$...963bf1| 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 64 31 2d 30 35 36 61 2d 34 34 31 61 2d 62 37 62 |d1-056a-441a-b7b| 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 36 2d 34 38 30 62 34 39 32 34 61 36 63 32 08 00 |6-480b4924a6c2..| 2024-04-13T13:43:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 36 63 39 66 30 33 34 61 |....$...6c9f034a| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 35 65 31 63 2d 34 39 32 61 2d 61 63 63 35 2d |-5e1c-492a-acc5-| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 64 66 30 63 63 32 31 35 32 64 64 33 08 8c 1a 66 |df0cc2152dd3...f| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 bf e0 85 0f |....| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 38 35 36 31 36 32 36 31 2d 64 31 37 37 2d 34 35 |85616261-d177-45| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 32 2d 61 34 63 34 2d 64 62 34 30 61 30 35 63 |82-a4c4-db40a05c| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 37 33 66 |273f| 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2024-04-13T13:43:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent rbd_mirror_leader omap vals' 2024-04-13T13:43:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2024-04-13T13:43:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2024-04-13T13:43:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_4322 2024-04-13T13:43:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_4849 2024-04-13T13:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:instance_5194 2024-04-13T13:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (0 bytes) : 2024-04-13T13:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T13:43:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns1 images 2024-04-13T13:43:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent ns1 images' 2024-04-13T13:43:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2024-04-13T13:43:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2024-04-13T13:43:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2024-04-13T13:43:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns1 mirror pool info 2024-04-13T13:43:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool info' 2024-04-13T13:43:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2024-04-13T13:43:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2024-04-13T13:43:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns1 mirror pool status 2024-04-13T13:43:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool status' 2024-04-13T13:43:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2024-04-13T13:43:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2024-04-13T13:43:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2024-04-13T13:43:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2024-04-13T13:43:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirroring omap vals' 2024-04-13T13:43:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2024-04-13T13:43:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2024-04-13T13:43:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirror_leader omap vals' 2024-04-13T13:43:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2024-04-13T13:43:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2024-04-13T13:43:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2024-04-13T13:43:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T13:43:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns2 images 2024-04-13T13:43:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent ns2 images' 2024-04-13T13:43:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2024-04-13T13:43:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2024-04-13T13:43:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2024-04-13T13:43:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns2 mirror pool info 2024-04-13T13:43:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool info' 2024-04-13T13:43:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2024-04-13T13:43:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2024-04-13T13:43:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns2 mirror pool status 2024-04-13T13:43:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool status' 2024-04-13T13:43:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2024-04-13T13:43:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2024-04-13T13:43:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2024-04-13T13:43:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2024-04-13T13:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirroring omap vals' 2024-04-13T13:43:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2024-04-13T13:43:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2024-04-13T13:43:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2024-04-13T13:43:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirror_leader omap vals' 2024-04-13T13:43:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2024-04-13T13:43:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2024-04-13T13:43:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:43:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:43:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-04-13T13:43:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 status 2024-04-13T13:43:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 status' 2024-04-13T13:43:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:43:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster2 -s 2024-04-13T13:48:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13T13:48:37.780+0000 7efd81496640 0 monclient(hunting): authenticate timed out after 300 2024-04-13T13:48:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-13T13:48:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:48:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster2 service dump 2024-04-13T13:48:42.990 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.1 2024-04-13T13:48:42.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 6 2024-04-13T13:48:42.991 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:*** Caught signal (Aborted) ** 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: in thread 7f5203021640 thread_name:rbd-mirror 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5202e54db0] 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f5202e9c39a] 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 3: pthread_cond_wait() 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f52032d5500] 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 5: rbd-mirror(+0x2ef7bb) [0x5635800ed7bb] 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0xe7) [0x5635800f6437] 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5635800f6752] 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 8: rbd-mirror(+0x8b21f8) [0x5635806b01f8] 2024-04-13T13:48:43.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 9: rbd-mirror(+0x8b21e0) [0x5635806b01e0] 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5635800e5c05] 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 11: main() 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f5202e3feb0] 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 13: __libc_start_main() 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 14: _start() 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:2024-04-13T13:48:43.003+0000 7f5203021640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: in thread 7f5203021640 thread_name:rbd-mirror 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5202e54db0] 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f5202e9c39a] 2024-04-13T13:48:43.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 3: pthread_cond_wait() 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f52032d5500] 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 5: rbd-mirror(+0x2ef7bb) [0x5635800ed7bb] 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0xe7) [0x5635800f6437] 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5635800f6752] 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 8: rbd-mirror(+0x8b21f8) [0x5635806b01f8] 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 9: rbd-mirror(+0x8b21e0) [0x5635806b01e0] 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5635800e5c05] 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 11: main() 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f5202e3feb0] 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 13: __libc_start_main() 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 14: _start() 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:48:43.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2024-04-13T13:48:43.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: -9995> 2024-04-13T13:38:36.859+0000 7f51ff7f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 56502) UID: 1000 2024-04-13T13:48:43.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 0> 2024-04-13T13:48:43.003+0000 7f5203021640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:48:43.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: in thread 7f5203021640 thread_name:rbd-mirror 2024-04-13T13:48:43.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2024-04-13T13:48:43.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:48:43.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5202e54db0] 2024-04-13T13:48:43.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f5202e9c39a] 2024-04-13T13:48:43.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 3: pthread_cond_wait() 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f52032d5500] 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 5: rbd-mirror(+0x2ef7bb) [0x5635800ed7bb] 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0xe7) [0x5635800f6437] 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5635800f6752] 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 8: rbd-mirror(+0x8b21f8) [0x5635806b01f8] 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 9: rbd-mirror(+0x8b21e0) [0x5635806b01e0] 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5635800e5c05] 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 11: main() 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f5202e3feb0] 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 13: __libc_start_main() 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 14: _start() 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:48:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2024-04-13T13:48:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: -9999> 2024-04-13T13:38:36.859+0000 7f51ff7f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 56502) UID: 1000 2024-04-13T13:48:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: -9998> 2024-04-13T13:48:43.003+0000 7f5203021640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:48:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: in thread 7f5203021640 thread_name:rbd-mirror 2024-04-13T13:48:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2024-04-13T13:48:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:48:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5202e54db0] 2024-04-13T13:48:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f5202e9c39a] 2024-04-13T13:48:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 3: pthread_cond_wait() 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f52032d5500] 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 5: rbd-mirror(+0x2ef7bb) [0x5635800ed7bb] 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0xe7) [0x5635800f6437] 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5635800f6752] 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 8: rbd-mirror(+0x8b21f8) [0x5635806b01f8] 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 9: rbd-mirror(+0x8b21e0) [0x5635806b01e0] 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5635800e5c05] 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 11: main() 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f5202e3feb0] 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 13: __libc_start_main() 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 14: _start() 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:48:43.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr: 2024-04-13T13:48:43.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi122.stderr:daemon-helper: command crashed with signal 6 2024-04-13T13:48:46.619 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2024-04-13T13:48:46.619 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-13T13:48:47.123 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2024-04-13T13:48:47.123 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2024-04-13T13:48:47.123 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2024-04-13T13:48:47.124 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2024-04-13T13:48:47.124 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2024-04-13T13:48:47.124 INFO:tasks.ceph.cluster1.osd.0.smithi083.stderr:2024-04-13T13:48:47.135+0000 7febf55f8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 33120) UID: 0 2024-04-13T13:48:47.125 INFO:tasks.ceph.cluster1.osd.0.smithi083.stderr:2024-04-13T13:48:47.135+0000 7febf55f8640 -1 osd.0 268 *** Got signal Terminated *** 2024-04-13T13:48:47.125 INFO:tasks.ceph.cluster1.osd.0.smithi083.stderr:2024-04-13T13:48:47.135+0000 7febf55f8640 -1 osd.0 268 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T13:48:47.125 INFO:tasks.ceph.cluster1.osd.1.smithi083.stderr:2024-04-13T13:48:47.135+0000 7fa831df8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 33123) UID: 0 2024-04-13T13:48:47.125 INFO:tasks.ceph.cluster1.osd.1.smithi083.stderr:2024-04-13T13:48:47.135+0000 7fa831df8640 -1 osd.1 268 *** Got signal Terminated *** 2024-04-13T13:48:47.125 INFO:tasks.ceph.cluster1.osd.1.smithi083.stderr:2024-04-13T13:48:47.135+0000 7fa831df8640 -1 osd.1 268 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T13:48:47.125 INFO:tasks.ceph.cluster1.osd.2.smithi083.stderr:2024-04-13T13:48:47.135+0000 7fa9877f8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 33126) UID: 0 2024-04-13T13:48:47.125 INFO:tasks.ceph.cluster1.osd.2.smithi083.stderr:2024-04-13T13:48:47.135+0000 7fa9877f8640 -1 osd.2 268 *** Got signal Terminated *** 2024-04-13T13:48:47.126 INFO:tasks.ceph.cluster1.osd.2.smithi083.stderr:2024-04-13T13:48:47.135+0000 7fa9877f8640 -1 osd.2 268 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T13:48:47.126 INFO:tasks.ceph.cluster1.mon.a.smithi083.stderr:2024-04-13T13:48:47.135+0000 7f0c9ebfc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 32788) UID: 0 2024-04-13T13:48:47.126 INFO:tasks.ceph.cluster1.mon.a.smithi083.stderr:2024-04-13T13:48:47.135+0000 7f0c9ebfc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-13T13:48:47.126 INFO:tasks.ceph.cluster1.mgr.x.smithi083.stderr:2024-04-13T13:48:47.135+0000 7f434e3f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 32790) UID: 0 2024-04-13T13:49:58.894 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.0 2024-04-13T13:49:58.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 6 2024-04-13T13:49:58.895 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-04-13T13:49:58.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr:*** Caught signal (Aborted) ** 2024-04-13T13:49:58.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: in thread 7f96b9c25640 thread_name:rbd-mirror 2024-04-13T13:49:58.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:49:58.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f96b9a54db0] 2024-04-13T13:49:58.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f96b9a9c39a] 2024-04-13T13:49:58.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 3: pthread_cond_wait() 2024-04-13T13:49:58.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f96b9ed5500] 2024-04-13T13:49:58.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 5: /lib64/librados.so.2(+0x13bc5b) [0x7f96bb766c5b] 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x557d45b66e48] 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x557d45b68cc8] 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x557d45b3de34] 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 9: main() 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f96b9a3feb0] 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 11: __libc_start_main() 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 12: _start() 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr:2024-04-13T13:49:58.908+0000 7f96b9c25640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: in thread 7f96b9c25640 thread_name:rbd-mirror 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:49:58.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f96b9a54db0] 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f96b9a9c39a] 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 3: pthread_cond_wait() 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f96b9ed5500] 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 5: /lib64/librados.so.2(+0x13bc5b) [0x7f96bb766c5b] 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x557d45b66e48] 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x557d45b68cc8] 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x557d45b3de34] 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 9: main() 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f96b9a3feb0] 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 11: __libc_start_main() 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 12: _start() 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:49:58.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2024-04-13T13:49:58.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: -1645> 2024-04-13T13:39:45.753+0000 7f96b63f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 58705) UID: 1000 2024-04-13T13:49:58.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 0> 2024-04-13T13:49:58.908+0000 7f96b9c25640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:49:58.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: in thread 7f96b9c25640 thread_name:rbd-mirror 2024-04-13T13:49:58.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2024-04-13T13:49:58.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:49:58.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f96b9a54db0] 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f96b9a9c39a] 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 3: pthread_cond_wait() 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f96b9ed5500] 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 5: /lib64/librados.so.2(+0x13bc5b) [0x7f96bb766c5b] 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x557d45b66e48] 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x557d45b68cc8] 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x557d45b3de34] 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 9: main() 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f96b9a3feb0] 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 11: __libc_start_main() 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 12: _start() 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:49:58.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2024-04-13T13:49:58.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: -8558> 2024-04-13T13:39:45.753+0000 7f96b63f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 58705) UID: 1000 2024-04-13T13:49:58.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: -8557> 2024-04-13T13:49:58.908+0000 7f96b9c25640 -1 *** Caught signal (Aborted) ** 2024-04-13T13:49:58.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: in thread 7f96b9c25640 thread_name:rbd-mirror 2024-04-13T13:49:58.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2024-04-13T13:49:58.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable) 2024-04-13T13:49:58.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f96b9a54db0] 2024-04-13T13:49:58.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f96b9a9c39a] 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 3: pthread_cond_wait() 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f96b9ed5500] 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 5: /lib64/librados.so.2(+0x13bc5b) [0x7f96bb766c5b] 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x557d45b66e48] 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x557d45b68cc8] 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x557d45b3de34] 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 9: main() 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f96b9a3feb0] 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 11: __libc_start_main() 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 12: _start() 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-13T13:49:58.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr: 2024-04-13T13:49:59.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi122.stderr:daemon-helper: command crashed with signal 6 2024-04-13T13:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13T13:53:37.870+0000 7fa9380c2640 0 monclient(hunting): authenticate timed out after 300 2024-04-13T13:53:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-13T13:53:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T13:53:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster2 service status 2024-04-13T13:58:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13T13:58:37.955+0000 7f34222dc640 0 monclient(hunting): authenticate timed out after 300 2024-04-13T13:58:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-13T13:58:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T13:58:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T13:58:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-04-13T13:58:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T13:58:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror images' 2024-04-13T13:58:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror images 2024-04-13T13:58:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2024-04-13T14:03:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T14:03:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (110) Connection timed out 2024-04-13T14:03:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T14:03:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T14:03:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror mirror pool info' 2024-04-13T14:03:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror mirror pool info 2024-04-13T14:03:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2024-04-13T14:08:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T14:08:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T14:08:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T14:08:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror mirror pool status 2024-04-13T14:08:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror mirror pool status' 2024-04-13T14:08:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T14:08:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2024-04-13T14:13:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T14:13:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T14:13:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T14:13:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls 2024-04-13T14:18:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror rbd_mirroring omap vals 2024-04-13T14:18:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror rbd_mirroring omap vals' 2024-04-13T14:18:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2024-04-13T14:23:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T14:23:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror rbd_mirror_leader omap vals 2024-04-13T14:23:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2024-04-13T14:23:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2024-04-13T14:28:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T14:28:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T14:28:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T14:28:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T14:28:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns1 images 2024-04-13T14:28:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror ns1 images' 2024-04-13T14:28:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2024-04-13T14:33:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T14:33:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (110) Connection timed out 2024-04-13T14:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T14:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T14:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns1 mirror pool info 2024-04-13T14:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirrorns1 mirror pool info' 2024-04-13T14:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2024-04-13T14:38:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T14:38:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T14:38:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T14:38:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns1 mirror pool status 2024-04-13T14:38:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirrorns1 mirror pool status' 2024-04-13T14:38:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T14:38:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2024-04-13T14:43:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T14:43:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T14:43:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T14:43:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2024-04-13T14:48:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2024-04-13T14:48:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2024-04-13T14:48:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2024-04-13T14:53:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T14:53:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2024-04-13T14:53:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2024-04-13T14:53:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2024-04-13T14:58:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T14:58:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T14:58:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T14:58:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T14:58:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns2 images 2024-04-13T14:58:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror ns2 images' 2024-04-13T14:58:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2024-04-13T15:03:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T15:03:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (110) Connection timed out 2024-04-13T15:03:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T15:03:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T15:03:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns2 mirror pool info 2024-04-13T15:03:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirrorns2 mirror pool info' 2024-04-13T15:03:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2024-04-13T15:08:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T15:08:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T15:08:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T15:08:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns2 mirror pool status 2024-04-13T15:08:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirrorns2 mirror pool status' 2024-04-13T15:08:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T15:08:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2024-04-13T15:13:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T15:13:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T15:13:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T15:13:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2024-04-13T15:18:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2024-04-13T15:18:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2024-04-13T15:18:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2024-04-13T15:23:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T15:23:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2024-04-13T15:23:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2024-04-13T15:23:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2024-04-13T15:28:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T15:28:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T15:28:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T15:28:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-04-13T15:28:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T15:28:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent images 2024-04-13T15:28:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent images' 2024-04-13T15:28:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2024-04-13T15:33:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T15:33:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (110) Connection timed out 2024-04-13T15:33:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T15:33:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T15:33:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent mirror pool info 2024-04-13T15:33:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent mirror pool info' 2024-04-13T15:33:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2024-04-13T15:38:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T15:38:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T15:38:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent mirror pool status 2024-04-13T15:38:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T15:38:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent mirror pool status' 2024-04-13T15:38:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T15:38:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2024-04-13T15:43:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T15:43:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T15:43:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T15:43:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2024-04-13T15:48:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2024-04-13T15:48:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2024-04-13T15:48:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2024-04-13T15:53:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T15:53:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2024-04-13T15:53:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2024-04-13T15:53:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2024-04-13T15:58:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T15:58:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T15:58:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T15:58:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T15:58:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns1 images 2024-04-13T15:58:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent ns1 images' 2024-04-13T15:58:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2024-04-13T16:03:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T16:03:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (110) Connection timed out 2024-04-13T16:03:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T16:03:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T16:03:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns1 mirror pool info 2024-04-13T16:03:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool info' 2024-04-13T16:03:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2024-04-13T16:08:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T16:08:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T16:08:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T16:08:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns1 mirror pool status 2024-04-13T16:08:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool status' 2024-04-13T16:08:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T16:08:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2024-04-13T16:13:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T16:13:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T16:13:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T16:13:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2024-04-13T16:18:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2024-04-13T16:18:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirroring omap vals' 2024-04-13T16:18:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2024-04-13T16:23:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T16:23:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns1 rbd_mirror_leader omap vals 2024-04-13T16:23:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirror_leader omap vals' 2024-04-13T16:23:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2024-04-13T16:28:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T16:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T16:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T16:28:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-04-13T16:28:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns2 images 2024-04-13T16:28:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent ns2 images' 2024-04-13T16:28:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2024-04-13T16:33:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T16:33:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (110) Connection timed out 2024-04-13T16:33:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T16:33:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T16:33:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns2 mirror pool info 2024-04-13T16:33:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool info' 2024-04-13T16:33:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2024-04-13T16:38:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T16:38:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T16:38:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T16:38:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns2 mirror pool status 2024-04-13T16:38:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool status' 2024-04-13T16:38:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T16:38:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2024-04-13T16:43:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: couldn't connect to the cluster! 2024-04-13T16:43:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T16:43:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T16:43:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2024-04-13T16:48:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2024-04-13T16:48:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirroring omap vals' 2024-04-13T16:48:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2024-04-13T16:53:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T16:53:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns2 rbd_mirror_leader omap vals 2024-04-13T16:53:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirror_leader omap vals' 2024-04-13T16:53:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2024-04-13T16:58:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-13T16:58:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2024-04-13T16:58:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2024-04-13T16:58:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret 2024-04-13T16:58:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-13T16:58:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 0 3 2024-04-13T16:58:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T16:58:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ daemon_pid_file cluster1:0 2024-04-13T16:58:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:0 2024-04-13T16:58:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T16:58:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:0 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T16:58:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-04-13T16:58:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-04-13T16:58:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-04-13T16:58:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ']' 2024-04-13T16:58:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist) 2024-04-13T16:58:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist)' 2024-04-13T16:58:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ daemon_pid_file cluster1:1 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:1 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:1 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T16:58:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T16:58:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T16:58:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T16:58:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-04-13T16:58:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-04-13T16:58:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-04-13T16:58:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ']' 2024-04-13T16:58:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist)' 2024-04-13T16:58:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist) 2024-04-13T16:58:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2024-04-13T16:58:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ daemon_pid_file cluster1:2 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:2 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:2 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T16:58:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T16:58:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2024-04-13T16:58:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2024-04-13T16:58:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2024-04-13T16:58:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid ']' 2024-04-13T16:58:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist)' 2024-04-13T16:58:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist) 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ daemon_pid_file cluster1:3 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster1:3 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster1:3 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster1 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-13T16:58:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster1 2024-04-13T16:58:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster1 2024-04-13T16:58:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T16:58:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T16:58:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2024-04-13T16:58:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2024-04-13T16:58:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2024-04-13T16:58:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid ']' 2024-04-13T16:58:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist)' 2024-04-13T16:58:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist) 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 0 3 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ daemon_pid_file cluster2:0 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster2:0 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster2:0 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster2 2024-04-13T16:58:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=0 2024-04-13T16:58:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 0 = cluster2:0 ']' 2024-04-13T16:58:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster2 2024-04-13T16:58:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster2 2024-04-13T16:58:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=0 2024-04-13T16:58:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=0 2024-04-13T16:58:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-04-13T16:58:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-04-13T16:58:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-04-13T16:58:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid ']' 2024-04-13T16:58:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist)' 2024-04-13T16:58:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist) 2024-04-13T16:58:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2024-04-13T16:58:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T16:58:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ daemon_pid_file cluster2:1 2024-04-13T16:58:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster2:1 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster2:1 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster2 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=1 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 1 = cluster2:1 ']' 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster2 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster2 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=1 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=1 2024-04-13T16:58:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-04-13T16:58:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-04-13T16:58:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-04-13T16:58:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid ']' 2024-04-13T16:58:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist)' 2024-04-13T16:58:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist) 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ daemon_pid_file cluster2:2 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster2:2 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster2:2 cluster instance 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster2:2 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T16:58:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T16:58:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster2 2024-04-13T16:58:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=2 2024-04-13T16:58:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 2 = cluster2:2 ']' 2024-04-13T16:58:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster2 2024-04-13T16:58:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster2 2024-04-13T16:58:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=2 2024-04-13T16:58:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=2 2024-04-13T16:58:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2024-04-13T16:58:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2024-04-13T16:58:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2024-04-13T16:58:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid ']' 2024-04-13T16:58:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist)' 2024-04-13T16:58:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist) 2024-04-13T16:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2024-04-13T16:58:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ daemon_pid_file cluster2:3 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster=cluster2:3 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ set_cluster_instance cluster2:3 cluster instance 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local val=cluster2:3 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local cluster_var_name=cluster 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ local instance_var_name=instance 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ cluster=cluster2 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ instance=3 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ '[' 3 = cluster2:3 ']' 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval cluster=cluster2 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ cluster=cluster2 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ eval instance=3 2024-04-13T16:58:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ instance=3 2024-04-13T16:58:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2024-04-13T16:58:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2024-04-13T16:58:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2024-04-13T16:58:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid ']' 2024-04-13T16:58:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist)' 2024-04-13T16:58:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist) 2024-04-13T16:58:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2024-04-13T16:58:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T16:58:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -z '' ']' 2024-04-13T16:58:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster instance 2024-04-13T16:58:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T16:58:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-13T17:03:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13T17:03:39.317+0000 7fa6ae84c640 0 monclient(hunting): authenticate timed out after 300 2024-04-13T17:03:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-13T17:03:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T17:03:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-13T17:08:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13T17:08:39.409+0000 7f71c211c640 0 monclient(hunting): authenticate timed out after 300 2024-04-13T17:08:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-13T17:08:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T17:08:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-13T17:13:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13T17:13:39.495+0000 7f0045b1c640 0 monclient(hunting): authenticate timed out after 300 2024-04-13T17:13:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-13T17:13:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= 2024-04-13T17:13:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-13T17:18:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2024-04-13T17:18:39.583+0000 7f5a5b12a640 0 monclient(hunting): authenticate timed out after 300 2024-04-13T17:18:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-13T17:18:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-13T17:18:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirrors cluster1 2024-04-13T17:18:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2024-04-13T17:18:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 0 3 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:0 '' 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:1 '' 2024-04-13T17:18:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:2 '' 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:3 '' 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2024-04-13T17:18:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirrors cluster2 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:++ seq 0 3 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster2:0 '' 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T17:18:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster2:1 '' 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster2:2 '' 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-13T17:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster2:3 '' 2024-04-13T17:18:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2024-04-13T17:18:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2024-04-13T17:18:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 1 2024-04-13T17:18:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 2024-04-13T17:18:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' -z 1 ']' 2024-04-13T17:18:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test '' = /tmp/tmp.6pyizPAJcx 2024-04-13T17:18:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -Rf /tmp/tmp.6pyizPAJcx 2024-04-13T17:18:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:FAIL 2024-04-13T17:18:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-13T17:18:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ '[' 110 -eq 0 ']' 2024-04-13T17:18:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo FAIL 2024-04-13T17:18:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ exit 110 2024-04-13T17:18:39.596 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-04-13T17:18:39.596 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-04-13T17:18:39.887 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi122 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f70a9a90881107996c8b7a3e89ba8b32624713c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2024-04-13T17:18:40.105 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=18c520492b214a8f9a52dfefb7e684d7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi122 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f70a9a90881107996c8b7a3e89ba8b32624713c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2024-04-13T17:18:40.108 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-13T17:18:40.121 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-13T17:18:40.121 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-04-13T17:18:40.162 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-13T17:18:40.173 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-13T17:18:40.173 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 211, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-04-13T17:18:40.178 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-13T17:18:40.207 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-04-13T17:18:40.207 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:18:40.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi122.stderr:2024-04-13T17:18:40.207+0000 7f1dd13f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 45977) UID: 1000 2024-04-13T17:23:39.279 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T17:23:39.327 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-13T17:23:39.341 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-04-13T17:23:39.342 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:23:39.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi122.stderr:2024-04-13T17:23:39.342+0000 7f06b85f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 52784) UID: 1000 2024-04-13T17:28:38.415 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T17:28:38.421 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-13T17:28:38.433 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-04-13T17:28:38.433 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:28:38.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi122.stderr:2024-04-13T17:28:38.433+0000 7f46887f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 43110) UID: 1000 2024-04-13T17:33:37.510 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T17:33:37.516 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-13T17:33:37.577 ERROR:tasks.rbd_mirror.cluster2.client.mirror.0:tried to stop a non-running daemon 2024-04-13T17:33:37.581 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-13T17:33:37.592 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-04-13T17:33:37.592 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:33:37.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi122.stderr:2024-04-13T17:33:37.592+0000 7fcf53bf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 41313) UID: 1000 2024-04-13T17:38:36.658 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T17:38:36.685 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-13T17:38:36.698 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-04-13T17:38:36.699 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:38:36.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi122.stderr:2024-04-13T17:38:36.699+0000 7fb8947f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 53907) UID: 1000 2024-04-13T17:43:35.762 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T17:43:35.768 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-13T17:43:35.781 ERROR:tasks.rbd_mirror.cluster1.client.mirror.1:tried to stop a non-running daemon 2024-04-13T17:43:35.784 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-13T17:43:35.938 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-04-13T17:43:35.938 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:43:35.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi122.stderr:2024-04-13T17:43:35.939+0000 7fc7917f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 58776) UID: 1000 2024-04-13T17:48:35.002 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T17:48:35.028 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-13T17:48:35.042 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-13T17:48:35.042 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-13T17:50:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-13T17:50:35.087 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1934, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-04-13T17:50:35.128 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-13T17:50:35.128 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-13T17:50:35.128 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-04-13T17:50:35.128 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:50:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-04-13T17:50:35.134 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1934, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2024-04-13T17:50:35.149 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-04-13T17:50:35.149 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-04-13T17:50:35.149 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:50:35.150 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-04-13T17:50:35.150 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-04-13T17:50:35.150 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:50:35.150 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-04-13T17:50:35.150 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-13T17:50:35.151 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-04-13T17:50:35.151 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:50:35.151 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-04-13T17:50:35.151 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-13T17:50:35.151 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-04-13T17:50:35.151 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:50:35.152 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-04-13T17:50:35.152 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-13T17:50:35.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-13T17:50:35.190 INFO:teuthology.orchestra.run.smithi122.stdout:2024-04-13T13:33:23.832993+0000 mon.a (mon.0) 813 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-13T17:50:35.190 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-13T17:50:35.191 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T17:50:35.191 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-04-13T17:50:35.494 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T17:50:35.494 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-04-13T17:50:35.662 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T17:50:35.662 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-04-13T17:50:35.822 INFO:tasks.ceph:Archiving mon data... 2024-04-13T17:50:35.824 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/data/cluster2.mon.a.tgz 2024-04-13T17:50:35.824 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-04-13T17:50:35.838 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.TLS8mdqDYb 2024-04-13T17:50:35.839 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.TLS8mdqDYb 2024-04-13T17:50:36.003 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.TLS8mdqDYb 2024-04-13T17:50:36.088 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.TLS8mdqDYb is 448KB 2024-04-13T17:50:36.186 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.TLS8mdqDYb 2024-04-13T17:50:36.200 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-13T17:50:36.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-13T17:50:36.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-13T17:50:36.267 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1934, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-04-13T17:50:36.281 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-13T17:50:36.297 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-13T17:50:36.297 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-13T17:52:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-13T17:52:36.338 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1934, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2024-04-13T17:52:36.353 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-13T17:52:36.354 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-13T17:52:36.354 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-04-13T17:52:36.354 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:52:36.358 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-04-13T17:52:36.358 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-04-13T17:52:36.358 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:52:36.358 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-04-13T17:52:36.358 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-04-13T17:52:36.359 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:52:36.359 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-04-13T17:52:36.359 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-13T17:52:36.359 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-04-13T17:52:36.359 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:52:36.359 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-04-13T17:52:36.360 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-13T17:52:36.360 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-04-13T17:52:36.360 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T17:52:36.360 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-04-13T17:52:36.360 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-13T17:52:36.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-13T17:52:36.448 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T17:52:36.448 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-04-13T17:52:36.510 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T17:52:36.510 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-04-13T17:52:36.565 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T17:52:36.565 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-04-13T17:52:36.632 INFO:tasks.ceph:Archiving mon data... 2024-04-13T17:52:36.633 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/data/cluster1.mon.a.tgz 2024-04-13T17:52:36.633 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-04-13T17:52:36.652 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.HprOUYKAqn 2024-04-13T17:52:36.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.HprOUYKAqn 2024-04-13T17:52:36.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.HprOUYKAqn 2024-04-13T17:52:36.906 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.HprOUYKAqn is 365KB 2024-04-13T17:52:37.003 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.HprOUYKAqn 2024-04-13T17:52:37.017 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-13T17:52:37.018 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-13T17:52:37.059 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-13T17:52:37.092 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-13T17:52:37.092 DEBUG:teuthology.orchestra.run.smithi083:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-13T17:52:37.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-13T17:52:37.152 INFO:teuthology.orchestra.run.smithi083.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-13T17:52:37.164 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-13T17:52:37.169 INFO:tasks.ceph:Archiving crash dumps... 2024-04-13T17:52:37.171 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/remote/smithi083/crash 2024-04-13T17:52:37.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-13T17:52:37.224 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/remote/smithi122/crash 2024-04-13T17:52:37.225 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-13T17:52:37.302 INFO:tasks.ceph:Compressing logs... 2024-04-13T17:52:37.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T17:52:37.305 DEBUG:teuthology.orchestra.run.smithi122:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T17:52:38.488 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.41330.log: file size changed while zipping 2024-04-13T17:55:23.756 INFO:tasks.ceph:Archiving logs... 2024-04-13T17:55:23.757 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/remote/smithi083/log 2024-04-13T17:55:23.758 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-13T17:55:26.244 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/remote/smithi122/log 2024-04-13T17:55:26.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-13T17:55:46.885 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1934, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2024-04-13T17:55:46.891 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-13T17:55:46.910 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-04-13T17:55:46.918 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-13T17:55:46.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-13T17:55:46.933 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-13T17:55:47.018 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-13T17:55:47.030 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-13T17:55:47.030 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T17:55:47.033 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T17:55:47.046 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-04-13T17:55:47.049 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-13T17:55:47.051 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T17:55:47.051 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-04-13T17:55:47.051 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv01.front.sepia.ceph.com 3 6 377 58 +1314us[+1572us] +/- 39ms 2024-04-13T17:55:47.051 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 -2161us[-2161us] +/- 61ms 2024-04-13T17:55:47.051 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +393us[ +393us] +/- 62ms 2024-04-13T17:55:47.052 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T17:55:47.053 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T17:55:47.053 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-13T17:55:47.053 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 65 +1353us[+1605us] +/- 39ms 2024-04-13T17:55:47.053 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 7 377 17 -2076us[-1823us] +/- 62ms 2024-04-13T17:55:47.054 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +418us[ +418us] +/- 63ms 2024-04-13T17:55:47.054 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T17:55:47.054 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-13T17:55:47.066 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-13T17:55:47.066 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-13T17:55:47.078 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-13T17:55:47.163 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-13T17:55:47.236 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T17:55:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T17:55:47.268 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T17:55:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T17:55:47.299 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-13T17:55:47.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-13T17:55:47.324 INFO:teuthology.task.internal:Duration was 17464.784202 seconds 2024-04-13T17:55:47.324 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-13T17:55:47.336 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-13T17:55:47.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-13T17:55:47.339 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-13T17:55:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T17:55:47.384 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T17:55:47.594 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-13T17:55:47.594 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-04-13T17:55:47.595 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-13T17:55:47.654 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-04-13T17:55:47.654 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-13T17:55:47.674 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-13T17:55:47.675 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T17:55:47.697 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T17:55:47.743 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-13T17:55:47.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T17:55:47.764 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T17:55:47.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-13T17:55:47.926 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-13T17:55:47.927 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-13T17:55:47.954 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-13T17:55:47.981 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-13T17:55:47.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-13T17:55:47.996 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-13T17:55:48.019 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-04-13T17:55:48.046 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-04-13T17:55:48.085 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-13T17:55:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T17:55:48.099 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-13T17:55:48.139 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi122.front.sepia.ceph.com, flagging run as failed 2024-04-13T17:55:48.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-13T17:55:48.154 INFO:teuthology.task.internal:Transferring archived files... 2024-04-13T17:55:48.155 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/remote/smithi083 2024-04-13T17:55:48.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-13T17:55:48.192 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/remote/smithi122 2024-04-13T17:55:48.192 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-13T17:56:31.485 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-04-13T17:56:31.940 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008/remote/smithi122/coredump/1713016123.56519.core.gz: gzip compressed data, was "1713016123.56519.core", last modified: Sat Apr 13 17:55:48 2024, max compression, original size modulo 2^32 740646912 2024-04-13T17:56:31.940 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-04-13T17:56:35.111 INFO:teuthology.task.internal: core looks like: /tmp/tmprug_qylg: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'rbd-mirror --foreground --cluster cluster1 --id mirror.1', real uid: 1000, effective uid: 1000, real gid: 1301, effective gid: 1301, execfn: '/usr/bin/rbd-mirror', platform: 'x86_64' 2024-04-13T17:56:35.187 INFO:teuthology.task.internal: dump_program: rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-13T17:56:35.188 DEBUG:teuthology.orchestra.run.smithi122:> which 'rbd-mirror --foreground --cluster cluster1 --id mirror.1' 2024-04-13T17:56:35.207 INFO:teuthology.orchestra.run.smithi122.stderr:/usr/bin/which: unrecognized option '--foreground' 2024-04-13T17:56:35.207 INFO:teuthology.orchestra.run.smithi122.stderr:/usr/bin/which: unrecognized option '--cluster' 2024-04-13T17:56:35.208 INFO:teuthology.orchestra.run.smithi122.stderr:/usr/bin/which: unrecognized option '--id' 2024-04-13T17:56:35.208 INFO:teuthology.orchestra.run.smithi122.stderr:/usr/bin/which: no cluster1 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-13T17:56:35.208 INFO:teuthology.orchestra.run.smithi122.stderr:/usr/bin/which: no mirror.1 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-13T17:56:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-13T17:56:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:/usr/bin/rbd-mirror 2024-04-13T17:56:35.210 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 2: "which 'rbd-mirror --foreground --cluster cluster1 --id mirror.1'" 2024-04-13T17:56:35.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-13T17:56:35.243 INFO:teuthology.task.internal:Not uploading archives. 2024-04-13T17:56:35.243 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-13T17:56:35.298 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-13T17:56:35.298 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-13T17:56:35.305 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-13T17:56:35.320 INFO:teuthology.orchestra.run.smithi083.stdout: 265891 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 13 17:55 /home/ubuntu/cephtest 2024-04-13T17:56:35.320 INFO:teuthology.orchestra.run.smithi083.stdout: 265919 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 13 17:55 /home/ubuntu/cephtest/archive 2024-04-13T17:56:35.321 INFO:teuthology.orchestra.run.smithi083.stdout: 265947 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 17:55 /home/ubuntu/cephtest/archive/syslog 2024-04-13T17:56:35.321 INFO:teuthology.orchestra.run.smithi083.stdout: 265389 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 13 13:04 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-04-13T17:56:35.321 INFO:teuthology.orchestra.run.smithi083.stdout: 265809 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 13 13:04 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-04-13T17:56:35.321 INFO:teuthology.orchestra.run.smithi083.stdout: 265950 184 -rw-r--r-- 1 ubuntu ubuntu 184546 Apr 13 17:55 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T17:56:35.321 INFO:teuthology.orchestra.run.smithi083.stdout: 400381 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 17:55 /home/ubuntu/cephtest/archive/audit 2024-04-13T17:56:35.321 INFO:teuthology.orchestra.run.smithi083.stdout: 400661 116 -rw------- 1 ubuntu ubuntu 118304 Apr 13 17:55 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-13T17:56:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T17:56:35.323 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-13T17:56:35.323 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_4f70a9a90881107996c8b7a3e89ba8b32624713c/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-13T17:56:35.333 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-13T17:56:35.349 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-13T17:56:35.362 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during thrashing 2024-04-13T17:56:35.362 INFO:teuthology.nuke:Checking targets against current locks 2024-04-13T17:56:35.369 INFO:teuthology.orchestra.run.smithi122.stdout: 265478 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 13 17:55 /home/ubuntu/cephtest 2024-04-13T17:56:35.369 INFO:teuthology.orchestra.run.smithi122.stdout: 402750 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 13 13:12 /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-13T17:56:35.369 INFO:teuthology.orchestra.run.smithi122.stdout: 402751 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 13 13:13 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-13T17:56:35.369 INFO:teuthology.orchestra.run.smithi122.stdout: 415112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 13:13 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-04-13T17:56:35.369 INFO:teuthology.orchestra.run.smithi122.stdout: 265525 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Apr 13 17:55 /home/ubuntu/cephtest/archive 2024-04-13T17:56:35.369 INFO:teuthology.orchestra.run.smithi122.stdout: 265570 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 17:55 /home/ubuntu/cephtest/archive/syslog 2024-04-13T17:56:35.369 INFO:teuthology.orchestra.run.smithi122.stdout: 265389 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 13 13:04 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-04-13T17:56:35.370 INFO:teuthology.orchestra.run.smithi122.stdout: 265600 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 13 13:04 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-04-13T17:56:35.370 INFO:teuthology.orchestra.run.smithi122.stdout: 265595 184 -rw-r--r-- 1 ubuntu ubuntu 188081 Apr 13 17:55 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T17:56:35.370 INFO:teuthology.orchestra.run.smithi122.stdout: 402748 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 17:55 /home/ubuntu/cephtest/archive/audit 2024-04-13T17:56:35.370 INFO:teuthology.orchestra.run.smithi122.stdout: 402752 124 -rw------- 1 ubuntu ubuntu 125637 Apr 13 17:55 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-13T17:56:35.370 INFO:teuthology.orchestra.run.smithi122.stdout: 265539 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 13:49 /home/ubuntu/cephtest/archive/coredump 2024-04-13T17:56:35.370 INFO:teuthology.orchestra.run.smithi122.stdout: 268373 65064 -rw------- 1 ubuntu ubuntu 740646912 Apr 13 13:48 /home/ubuntu/cephtest/archive/coredump/1713016123.56519.core 2024-04-13T17:56:35.370 INFO:teuthology.orchestra.run.smithi122.stdout: 268374 52480 -rw------- 1 ubuntu ubuntu 640823296 Apr 13 13:49 /home/ubuntu/cephtest/archive/coredump/1713016198.58722.core 2024-04-13T17:56:35.371 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-13T17:56:35.394 DEBUG:teuthology.nuke:shortname: smithi083 2024-04-13T17:56:35.394 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T17:56:35.419 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 12:54:34.160147', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG//M3DRroy6RXlxQxY3rAUPaE9ctyve7QrEx+XePnci2VHICPl1SRaHiQ5hVKBv5+qaWfUZ/wjpNTtD3aY7/4U='} 2024-04-13T17:56:35.422 DEBUG:teuthology.nuke:shortname: smithi122 2024-04-13T17:56:35.422 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T17:56:35.442 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-09_22:16:02-rbd-reef-distro-default-smithi/7650008', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 12:54:34.162148', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRJMbsaAkfeSQKoa4lOc+m7Cn7k5Jn7V7UwZkRynUaEfDiUib85HyYAqrj1IrJgBro/SyCaVsnB0Pgix2dZxZ8='} 2024-04-13T17:56:35.468 INFO:teuthology.orchestra.console.smithi083:Power off 2024-04-13T17:56:35.469 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-13T17:56:35.495 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-04-13T17:56:35.495 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T17:56:35.514 INFO:teuthology.orchestra.console.smithi122:Power off 2024-04-13T17:56:35.514 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-13T17:56:35.524 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-04-13T17:56:35.538 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-04-13T17:56:35.538 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T17:56:35.569 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-04-13T17:56:39.525 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T17:56:39.571 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T17:56:39.637 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-04-13T17:56:39.681 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-04-13T17:56:43.640 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T17:56:43.682 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T17:56:43.751 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-04-13T17:56:43.792 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-04-13T17:56:43.853 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-04-13T17:56:43.893 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-04-13T17:56:44.000 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 17464.784202098846 failure_reason: 'Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi122 with status 110: ''mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f70a9a90881107996c8b7a3e89ba8b32624713c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\''\'' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=18c520492b214a8f9a52dfefb7e684d7 status: fail success: false 2024-04-13T17:56:44.000 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-13T17:56:44.084 INFO:teuthology.run:FAIL